Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251029 08:51:57.731063 416 test_util.cc:276] Using random seed: 1052388554
2025-10-29T08:51:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-29T08:51:57Z Disabled control of system clock
I20251029 08:51:57.768888 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 08:51:58.119096 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:51:58.119431 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:51:58.119510 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:51:58.128106 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 08:51:58.128219 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:51:58.128273 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 08:51:58.128319 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 08:51:58.138662 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:51:58.140429 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:51:58.142316 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 08:51:58.154373 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:58.154608 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:58.155273 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:51:58.156204 424 server_base.cc:1047] running on GCE node
I20251029 08:51:58.157260 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:51:58.159471 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:51:58.160857 424 hybrid_clock.cc:648] HybridClock initialized: now 1761727918160765 us; error 87 us; skew 500 ppm
I20251029 08:51:58.161370 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:51:58.164342 424 webserver.cc:492] Webserver started at http://127.0.104.62:37745/ using document root <none> and password file <none>
I20251029 08:51:58.165335 424 fs_manager.cc:362] Metadata directory not provided
I20251029 08:51:58.165474 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:51:58.165930 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:51:58.169764 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/instance:
uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.171075 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal/instance:
uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.178887 424 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20251029 08:51:58.183712 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:51:58.185849 424 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251029 08:51:58.186043 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.186272 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:51:58.215832 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:51:58.216847 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:51:58.217161 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:51:58.239634 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40435
I20251029 08:51:58.239743 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40435 every 8 connection(s)
I20251029 08:51:58.241478 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
I20251029 08:51:58.246253 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:51:58.247642 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 424
I20251029 08:51:58.247912 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal/instance
I20251029 08:51:58.250454 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38825
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 08:51:58.252487 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:58.270061 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
W20251029 08:51:58.271193 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
W20251029 08:51:58.272763 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38825: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
I20251029 08:51:58.322151 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } attempt: 1
W20251029 08:51:58.324339 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38825: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
I20251029 08:51:58.392630 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } attempt: 2
W20251029 08:51:58.394562 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38825: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
I20251029 08:51:58.485836 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } attempt: 3
W20251029 08:51:58.487859 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38825: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
W20251029 08:51:58.604326 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:51:58.604665 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:51:58.604746 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:51:58.613581 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 08:51:58.613695 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:51:58.613749 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 08:51:58.613794 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 08:51:58.623852 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:51:58.625593 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:51:58.627439 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 08:51:58.631168 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } attempt: 4
W20251029 08:51:58.633138 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38825: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38825: connect: Connection refused (error 111)
W20251029 08:51:58.637287 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:58.637780 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:58.637789 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:51:58.638228 492 server_base.cc:1047] running on GCE node
I20251029 08:51:58.638983 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:51:58.641033 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:51:58.642259 492 hybrid_clock.cc:648] HybridClock initialized: now 1761727918642210 us; error 55 us; skew 500 ppm
I20251029 08:51:58.642621 492 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:51:58.644901 492 webserver.cc:492] Webserver started at http://127.0.104.61:36163/ using document root <none> and password file <none>
I20251029 08:51:58.645686 492 fs_manager.cc:362] Metadata directory not provided
I20251029 08:51:58.645794 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:51:58.646178 492 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:51:58.648818 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data/instance:
uuid: "85c8cdf28bda444d851fb50588400173"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.650210 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal/instance:
uuid: "85c8cdf28bda444d851fb50588400173"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.655936 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251029 08:51:58.659614 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:51:58.661070 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 08:51:58.661244 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal
uuid: "85c8cdf28bda444d851fb50588400173"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:51:58.661470 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:51:58.690361 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:51:58.691318 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:51:58.691601 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:51:58.713351 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:38825
I20251029 08:51:58.713369 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:38825 every 8 connection(s)
I20251029 08:51:58.714892 492 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/data/info.pb
I20251029 08:51:58.719264 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:51:58.720757 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 492
I20251029 08:51:58.720929 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-1/wal/instance
I20251029 08:51:58.721961 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42135
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 08:51:58.725531 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:58.743566 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:58.753156 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } has no permanent_uuid. Determining permanent_uuid...
W20251029 08:51:58.754208 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
W20251029 08:51:58.755769 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:58.805109 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 1
W20251029 08:51:58.807066 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:58.875317 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 2
W20251029 08:51:58.877427 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:58.932436 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } attempt: 5
I20251029 08:51:58.939481 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } has no permanent_uuid. Determining permanent_uuid...
W20251029 08:51:58.941465 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:58.968649 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 3
W20251029 08:51:58.970748 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:58.992720 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 1
W20251029 08:51:58.994659 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:59.062884 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 2
W20251029 08:51:59.064851 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
W20251029 08:51:59.087278 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:51:59.087654 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:51:59.087787 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:51:59.096532 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 08:51:59.096659 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:51:59.096743 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 08:51:59.096812 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 08:51:59.106822 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:51:59.108662 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:51:59.110481 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 08:51:59.114001 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 4
W20251029 08:51:59.115762 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
W20251029 08:51:59.119617 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:59.119585 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:51:59.120574 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:51:59.121186 563 server_base.cc:1047] running on GCE node
I20251029 08:51:59.121907 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:51:59.123296 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:51:59.124482 563 hybrid_clock.cc:648] HybridClock initialized: now 1761727919124427 us; error 47 us; skew 500 ppm
I20251029 08:51:59.124827 563 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:51:59.127449 563 webserver.cc:492] Webserver started at http://127.0.104.60:36171/ using document root <none> and password file <none>
I20251029 08:51:59.128274 563 fs_manager.cc:362] Metadata directory not provided
I20251029 08:51:59.128381 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:51:59.128741 563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:51:59.131098 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data/instance:
uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b"
format_stamp: "Formatted at 2025-10-29 08:51:59 on dist-test-slave-3g17"
I20251029 08:51:59.131877 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal/instance:
uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b"
format_stamp: "Formatted at 2025-10-29 08:51:59 on dist-test-slave-3g17"
I20251029 08:51:59.137295 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251029 08:51:59.140645 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:51:59.142015 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 08:51:59.142191 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal
uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b"
format_stamp: "Formatted at 2025-10-29 08:51:59 on dist-test-slave-3g17"
I20251029 08:51:59.142405 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:51:59.158102 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:51:59.159011 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:51:59.159269 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:51:59.171046 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 3
W20251029 08:51:59.173254 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42135: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42135: connect: Connection refused (error 111)
I20251029 08:51:59.181420 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:42135
I20251029 08:51:59.181437 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:42135 every 8 connection(s)
I20251029 08:51:59.182938 563 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/data/info.pb
I20251029 08:51:59.187215 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:51:59.190171 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 563
I20251029 08:51:59.190374 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-2/wal/instance
I20251029 08:51:59.193401 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:59.209684 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:59.217777 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } has no permanent_uuid. Determining permanent_uuid...
I20251029 08:51:59.230410 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Bootstrap starting.
I20251029 08:51:59.233665 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Neither blocks nor log segments found. Creating new log.
I20251029 08:51:59.234696 632 log.cc:826] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Log is configured to *not* fsync() on all Append() calls
I20251029 08:51:59.237037 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: No bootstrap required, opened a new log
I20251029 08:51:59.244657 632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.245011 632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:51:59.245126 632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8395ff5ec92c4bbda9bc6641a9fbbb6b, State: Initialized, Role: FOLLOWER
I20251029 08:51:59.245828 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.247121 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.247433 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.248333 632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: configured and running, proceeding with master startup.
W20251029 08:51:59.265184 652 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251029 08:51:59.265450 652 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251029 08:51:59.257205 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:40435: OK, instance_id { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" instance_seqno: 1761727918233350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 08:51:59.265836 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 08:51:59.299134 641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:51:59.299451 641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.301378 641 leader_election.cc:290] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825)
I20251029 08:51:59.350472 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 4
I20251029 08:51:59.364210 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap starting.
I20251029 08:51:59.368810 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Neither blocks nor log segments found. Creating new log.
I20251029 08:51:59.370327 490 log.cc:826] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Log is configured to *not* fsync() on all Append() calls
I20251029 08:51:59.374046 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: No bootstrap required, opened a new log
I20251029 08:51:59.382805 490 raft_consensus.cc:359] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.383210 490 raft_consensus.cc:385] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:51:59.383392 490 raft_consensus.cc:740] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f718bd0f00004f20a89d028a2fa8e6b9, State: Initialized, Role: FOLLOWER
I20251029 08:51:59.384241 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.385658 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.385972 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.386992 490 sys_catalog.cc:565] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 08:51:59.387061 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" is_pre_election: true
I20251029 08:51:59.387631 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in term 0.
I20251029 08:51:59.388401 583 leader_election.cc:304] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9; no voters:
I20251029 08:51:59.388926 641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:51:59.389096 641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:51:59.389228 641 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:51:59.392771 641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.394268 641 leader_election.cc:290] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 1 election: Requested vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825)
I20251029 08:51:59.395493 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
I20251029 08:51:59.395725 465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:51:59.399269 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in term 1.
I20251029 08:51:59.399838 583 leader_election.cc:304] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9; no voters:
I20251029 08:51:59.400203 641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:51:59.401309 641 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 LEADER]: Becoming Leader. State: Replica: 8395ff5ec92c4bbda9bc6641a9fbbb6b, State: Running, Role: LEADER
I20251029 08:51:59.401876 641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.406167 659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.406502 659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:51:59.407778 663 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20251029 08:51:59.411268 586 tablet.cc:2378] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:51:59.411798 663 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 08:51:59.414959 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } attempt: 5
I20251029 08:51:59.429304 465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Refusing update from remote peer 8395ff5ec92c4bbda9bc6641a9fbbb6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:51:59.430173 659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:51:59.433646 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: Bootstrap starting.
I20251029 08:51:59.447700 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: Neither blocks nor log segments found. Creating new log.
I20251029 08:51:59.450563 561 log.cc:826] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: Log is configured to *not* fsync() on all Append() calls
I20251029 08:51:59.453815 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.454056 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.458039 667 mvcc.cc:204] Tried to move back new op lower bound from 7216037557971165184 to 7216037557876375552. Current Snapshot: MvccSnapshot[applied={T|T < 7216037557971165184}]
I20251029 08:51:59.460435 670 mvcc.cc:204] Tried to move back new op lower bound from 7216037557971165184 to 7216037557876375552. Current Snapshot: MvccSnapshot[applied={T|T < 7216037557971165184}]
I20251029 08:51:59.463027 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: No bootstrap required, opened a new log
I20251029 08:51:59.471100 561 raft_consensus.cc:359] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.472198 561 raft_consensus.cc:385] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:51:59.472397 561 raft_consensus.cc:740] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c8cdf28bda444d851fb50588400173, State: Initialized, Role: FOLLOWER
I20251029 08:51:59.473239 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:51:59.479027 659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.479341 659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:51:59.484699 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.485169 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.486296 561 sys_catalog.cc:565] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 08:51:59.487185 534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:51:59.491590 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173"
I20251029 08:51:59.492842 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:51:59.494519 534 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Refusing update from remote peer 8395ff5ec92c4bbda9bc6641a9fbbb6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:51:59.495419 659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:51:59.495931 663 catalog_manager.cc:1357] Generated new cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:51:59.496052 663 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 08:51:59.518648 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.518921 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.531081 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:38825: OK, instance_id { permanent_uuid: "85c8cdf28bda444d851fb50588400173" instance_seqno: 1761727918707369 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 08:51:59.539261 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 08:51:59.542666 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.543037 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.543514 686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.543856 686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:51:59.550475 681 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:51:59.550582 681 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: loading cluster ID for follower catalog manager: success
W20251029 08:51:59.551843 681 catalog_manager.cc:1586] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20251029 08:51:59.551976 681 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20251029 08:51:59.553752 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:51:59.554044 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:51:59.560281 663 catalog_manager.cc:1380] Generated new certificate authority record
I20251029 08:51:59.562067 663 catalog_manager.cc:1514] Loading token signing keys...
I20251029 08:51:59.591439 693 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:51:59.591555 693 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: loading cluster ID for follower catalog manager: success
I20251029 08:51:59.592446 663 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Generated new TSK 0
I20251029 08:51:59.593871 663 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 08:51:59.595492 693 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: acquiring CA information for follower catalog manager: success
I20251029 08:51:59.597591 693 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 08:51:59.631839 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 08:52:00.104583 696 raft_consensus.cc:493] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:00.104888 696 raft_consensus.cc:515] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.106284 696 leader_election.cc:290] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:00.107170 697 raft_consensus.cc:493] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:00.107080 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:00.107363 697 raft_consensus.cc:515] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.108798 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" is_pre_election: true
I20251029 08:52:00.109045 697 leader_election.cc:290] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:00.109047 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85c8cdf28bda444d851fb50588400173 in term 1.
I20251029 08:52:00.110219 512 leader_election.cc:304] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85c8cdf28bda444d851fb50588400173, f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b
I20251029 08:52:00.111320 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:00.112941 534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:52:00.114136 696 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 08:52:00.114382 534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f718bd0f00004f20a89d028a2fa8e6b9 in term 1.
I20251029 08:52:00.115379 440 leader_election.cc:304] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85c8cdf28bda444d851fb50588400173, f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b
I20251029 08:52:00.115906 697 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 08:52:00.116065 697 raft_consensus.cc:493] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:00.116176 697 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 1 FOLLOWER]: Advancing to term 2
I20251029 08:52:00.119511 697 raft_consensus.cc:515] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.121287 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b"
I20251029 08:52:00.122218 697 leader_election.cc:290] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 2 election: Requested vote from peers 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:00.122887 534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173"
I20251029 08:52:00.123121 534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 1 FOLLOWER]: Advancing to term 2
I20251029 08:52:00.127094 534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f718bd0f00004f20a89d028a2fa8e6b9 in term 2.
I20251029 08:52:00.127820 440 leader_election.cc:304] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85c8cdf28bda444d851fb50588400173, f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b
I20251029 08:52:00.128425 697 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 FOLLOWER]: Leader election won for term 2
I20251029 08:52:00.128777 696 raft_consensus.cc:493] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:00.128952 696 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 2 FOLLOWER]: Advancing to term 3
I20251029 08:52:00.132241 696 raft_consensus.cc:515] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.136478 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
I20251029 08:52:00.138087 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b"
I20251029 08:52:00.139247 696 leader_election.cc:290] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 3 election: Requested vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:00.142179 697 raft_consensus.cc:697] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 LEADER]: Becoming Leader. State: Replica: f718bd0f00004f20a89d028a2fa8e6b9, State: Running, Role: LEADER
I20251029 08:52:00.142998 697 consensus_queue.cc:237] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.146456 464 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 LEADER]: Rejecting Update request from peer 8395ff5ec92c4bbda9bc6641a9fbbb6b for earlier term 1. Current term is 2. Ops: []
I20251029 08:52:00.147583 698 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 08:52:00.147631 512 leader_election.cc:304] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85c8cdf28bda444d851fb50588400173; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9
I20251029 08:52:00.148554 696 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251029 08:52:00.149220 699 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f718bd0f00004f20a89d028a2fa8e6b9. Latest consensus state: current_term: 2 leader_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:00.149680 699 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:00.156809 703 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 08:52:00.159732 703 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 08:52:00.160950 703 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:52:00.161142 703 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 08:52:00.165367 703 catalog_manager.cc:1514] Loading token signing keys...
I20251029 08:52:00.167757 703 catalog_manager.cc:6033] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Loaded TSK: 0
I20251029 08:52:00.169668 703 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251029 08:52:00.170550 695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:52:00.170951 695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:52:00.171105 695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251029 08:52:00.172942 698 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Peer f718bd0f00004f20a89d028a2fa8e6b9 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251029 08:52:00.174705 702 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 LEADER]: Stepping down as leader of term 1
I20251029 08:52:00.174853 702 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8395ff5ec92c4bbda9bc6641a9fbbb6b, State: Running, Role: LEADER
I20251029 08:52:00.175411 702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.176187 702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 1 FOLLOWER]: Advancing to term 2
W20251029 08:52:00.185155 695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:52:00.185436 695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251029 08:52:00.193559 534 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Rejecting Update request from peer 8395ff5ec92c4bbda9bc6641a9fbbb6b for earlier term 1. Current term is 3. Ops: []
I20251029 08:52:00.202003 695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:52:00.204628 695 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:52:00.208078 695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 08:52:00.221288 707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:00.222129 708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:00.226054 710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:00.226657 695 server_base.cc:1047] running on GCE node
I20251029 08:52:00.227329 695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:52:00.228911 695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:52:00.230147 695 hybrid_clock.cc:648] HybridClock initialized: now 1761727920230082 us; error 52 us; skew 500 ppm
I20251029 08:52:00.230480 695 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:52:00.233027 695 webserver.cc:492] Webserver started at http://127.0.104.1:32953/ using document root <none> and password file <none>
I20251029 08:52:00.233870 695 fs_manager.cc:362] Metadata directory not provided
I20251029 08:52:00.233983 695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:52:00.234372 695 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:52:00.236994 695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data/instance:
uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.237826 695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/instance:
uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.245713 695 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20251029 08:52:00.250726 716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.252641 695 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:00.252830 695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal
uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.253063 695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:52:00.267822 652 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 08:52:00.283396 695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:52:00.284368 695 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:52:00.284610 695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:52:00.285519 695 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 08:52:00.287175 695 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 08:52:00.287266 695 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.287421 695 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 08:52:00.287488 695 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.323714 695 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40969
I20251029 08:52:00.323771 828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40969 every 8 connection(s)
I20251029 08:52:00.325187 695 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/data/info.pb
I20251029 08:52:00.327672 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 695
I20251029 08:52:00.327879 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/instance
I20251029 08:52:00.337138 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 08:52:00.363844 831 heartbeater.cc:344] Connected to a master server at 127.0.104.61:38825
I20251029 08:52:00.363929 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:00.363874 829 heartbeater.cc:344] Connected to a master server at 127.0.104.60:42135
I20251029 08:52:00.364292 829 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.364288 830 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.364288 831 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.365433 829 heartbeater.cc:507] Master 127.0.104.60:42135 requested a full tablet report, sending...
I20251029 08:52:00.365489 830 heartbeater.cc:507] Master 127.0.104.62:40435 requested a full tablet report, sending...
I20251029 08:52:00.365933 831 heartbeater.cc:507] Master 127.0.104.61:38825 requested a full tablet report, sending...
I20251029 08:52:00.367861 455 ts_manager.cc:194] Registered new tserver with Master: 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:00.367861 526 ts_manager.cc:194] Registered new tserver with Master: 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:00.367908 597 ts_manager.cc:194] Registered new tserver with Master: 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:00.369820 455 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:49641
I20251029 08:52:00.557251 607 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Refusing update from remote peer f718bd0f00004f20a89d028a2fa8e6b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251029 08:52:00.558074 699 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 08:52:00.559926 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f718bd0f00004f20a89d028a2fa8e6b9. Latest consensus state: current_term: 2 leader_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:00.560245 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:00.562345 534 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Rejecting Update request from peer f718bd0f00004f20a89d028a2fa8e6b9 for earlier term 2. Current term is 3. Ops: []
I20251029 08:52:00.563503 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:00.563855 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:00.564062 697 consensus_queue.cc:1059] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 08:52:00.575394 699 raft_consensus.cc:3055] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 LEADER]: Stepping down as leader of term 2
I20251029 08:52:00.575621 699 raft_consensus.cc:740] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f718bd0f00004f20a89d028a2fa8e6b9, State: Running, Role: LEADER
I20251029 08:52:00.576145 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:00.576447 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:00.576088 699 consensus_queue.cc:260] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.577064 699 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 2 FOLLOWER]: Advancing to term 3
W20251029 08:52:00.704763 837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:52:00.705094 837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:52:00.705196 837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:52:00.713258 837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:52:00.713471 837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251029 08:52:00.723793 837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:52:00.725481 837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:52:00.727304 837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 08:52:00.738054 849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:00.738067 851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:00.738049 848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:00.738898 837 server_base.cc:1047] running on GCE node
I20251029 08:52:00.739518 837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:52:00.740536 837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:52:00.741746 837 hybrid_clock.cc:648] HybridClock initialized: now 1761727920741695 us; error 44 us; skew 500 ppm
I20251029 08:52:00.742055 837 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:52:00.744503 837 webserver.cc:492] Webserver started at http://127.0.104.2:35187/ using document root <none> and password file <none>
I20251029 08:52:00.745299 837 fs_manager.cc:362] Metadata directory not provided
I20251029 08:52:00.745407 837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:52:00.745769 837 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:52:00.748370 837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data/instance:
uuid: "927689e0dc5e4bc79d4a9d79a873f088"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.749222 837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/instance:
uuid: "927689e0dc5e4bc79d4a9d79a873f088"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.755430 837 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251029 08:52:00.758916 857 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.760532 837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 08:52:00.760706 837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal
uuid: "927689e0dc5e4bc79d4a9d79a873f088"
format_stamp: "Formatted at 2025-10-29 08:52:00 on dist-test-slave-3g17"
I20251029 08:52:00.760921 837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:52:00.782229 837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:52:00.783147 837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:52:00.783450 837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:52:00.784462 837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 08:52:00.786616 837 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 08:52:00.786722 837 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.786836 837 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 08:52:00.786906 837 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:00.825393 837 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42433
I20251029 08:52:00.825454 969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42433 every 8 connection(s)
I20251029 08:52:00.826894 970 raft_consensus.cc:493] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:00.827042 837 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/data/info.pb
I20251029 08:52:00.827064 970 raft_consensus.cc:515] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:00.827755 970 leader_election.cc:290] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:00.828428 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:00.828428 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85c8cdf28bda444d851fb50588400173" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" is_pre_election: true
I20251029 08:52:00.828794 464 raft_consensus.cc:2410] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85c8cdf28bda444d851fb50588400173 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251029 08:52:00.829371 512 leader_election.cc:304] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85c8cdf28bda444d851fb50588400173; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9
I20251029 08:52:00.829855 970 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251029 08:52:00.833884 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 837
I20251029 08:52:00.834095 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/instance
I20251029 08:52:00.844949 971 heartbeater.cc:344] Connected to a master server at 127.0.104.60:42135
I20251029 08:52:00.845124 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:00.845507 971 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.845504 972 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.845628 973 heartbeater.cc:344] Connected to a master server at 127.0.104.61:38825
I20251029 08:52:00.845772 973 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:00.846673 972 heartbeater.cc:507] Master 127.0.104.62:40435 requested a full tablet report, sending...
I20251029 08:52:00.846714 971 heartbeater.cc:507] Master 127.0.104.60:42135 requested a full tablet report, sending...
I20251029 08:52:00.846745 973 heartbeater.cc:507] Master 127.0.104.61:38825 requested a full tablet report, sending...
I20251029 08:52:00.848871 454 ts_manager.cc:194] Registered new tserver with Master: 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:00.849062 526 ts_manager.cc:194] Registered new tserver with Master: 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:00.849891 597 ts_manager.cc:194] Registered new tserver with Master: 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:00.872714 830 heartbeater.cc:499] Master 127.0.104.62:40435 was elected leader, sending a full tablet report...
I20251029 08:52:01.063078 705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f718bd0f00004f20a89d028a2fa8e6b9)
I20251029 08:52:01.063289 705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:01.063915 705 leader_election.cc:290] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825)
I20251029 08:52:01.064469 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" is_pre_election: true
I20251029 08:52:01.064563 534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:52:01.064728 464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in term 3.
I20251029 08:52:01.064905 534 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in current term 3: Already voted for candidate 85c8cdf28bda444d851fb50588400173 in this term.
I20251029 08:52:01.065244 583 leader_election.cc:304] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9; no voters:
I20251029 08:52:01.065553 705 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251029 08:52:01.065688 705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Starting leader election (detected failure of leader f718bd0f00004f20a89d028a2fa8e6b9)
I20251029 08:52:01.065789 705 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 2 FOLLOWER]: Advancing to term 3
I20251029 08:52:01.067978 705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:01.068708 705 leader_election.cc:290] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 3 election: Requested vote from peers f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435), 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825)
I20251029 08:52:01.069021 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
I20251029 08:52:01.069075 534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173"
I20251029 08:52:01.069299 534 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in current term 3: Already voted for candidate 85c8cdf28bda444d851fb50588400173 in this term.
I20251029 08:52:01.071151 464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8395ff5ec92c4bbda9bc6641a9fbbb6b in term 3.
I20251029 08:52:01.071506 583 leader_election.cc:304] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, f718bd0f00004f20a89d028a2fa8e6b9; no voters: 85c8cdf28bda444d851fb50588400173
I20251029 08:52:01.071763 705 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 3 FOLLOWER]: Leader election won for term 3
I20251029 08:52:01.072141 705 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [term 3 LEADER]: Becoming Leader. State: Replica: 8395ff5ec92c4bbda9bc6641a9fbbb6b, State: Running, Role: LEADER
I20251029 08:52:01.072393 705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:01.073359 978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.073544 978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:01.074043 979 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 08:52:01.075603 979 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 08:52:01.077087 979 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:52:01.077157 979 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 08:52:01.078281 979 catalog_manager.cc:1514] Loading token signing keys...
I20251029 08:52:01.078929 979 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Loaded TSK: 0
I20251029 08:52:01.079566 979 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 08:52:01.090502 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 08:52:01.353130 597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:58367
W20251029 08:52:01.456302 980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:52:01.456676 980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:52:01.456815 980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:52:01.465541 980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:52:01.465782 980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251029 08:52:01.475699 980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:52:01.477905 980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:52:01.479854 980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 08:52:01.490588 986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:01.490804 988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:01.490581 985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:01.491544 980 server_base.cc:1047] running on GCE node
I20251029 08:52:01.492187 980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:52:01.493153 980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:52:01.494365 980 hybrid_clock.cc:648] HybridClock initialized: now 1761727921494322 us; error 45 us; skew 500 ppm
I20251029 08:52:01.494663 980 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:52:01.496886 980 webserver.cc:492] Webserver started at http://127.0.104.3:32991/ using document root <none> and password file <none>
I20251029 08:52:01.497681 980 fs_manager.cc:362] Metadata directory not provided
I20251029 08:52:01.497797 980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:52:01.498148 980 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 08:52:01.500674 980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data/instance:
uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
format_stamp: "Formatted at 2025-10-29 08:52:01 on dist-test-slave-3g17"
I20251029 08:52:01.501525 980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/instance:
uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
format_stamp: "Formatted at 2025-10-29 08:52:01 on dist-test-slave-3g17"
I20251029 08:52:01.507396 980 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20251029 08:52:01.511011 994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:01.512554 980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 08:52:01.512725 980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal
uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
format_stamp: "Formatted at 2025-10-29 08:52:01 on dist-test-slave-3g17"
I20251029 08:52:01.512930 980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:52:01.522547 534 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [term 3 FOLLOWER]: Refusing update from remote peer 8395ff5ec92c4bbda9bc6641a9fbbb6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251029 08:52:01.523130 978 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 08:52:01.525980 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.526273 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:01.526093 999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.526422 999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:01.527205 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.527424 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:01.528386 464 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Refusing update from remote peer 8395ff5ec92c4bbda9bc6641a9fbbb6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251029 08:52:01.528497 999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.528754 999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cdf28bda444d851fb50588400173 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:01.529104 705 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251029 08:52:01.531309 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.531647 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:01.531229 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.531800 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:01.541679 980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:52:01.542589 980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:52:01.542834 980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:52:01.543767 980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 08:52:01.545514 980 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 08:52:01.545603 980 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:01.545683 980 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 08:52:01.545739 980 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:01.555259 681 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 08:52:01.582562 980 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:38031
I20251029 08:52:01.582633 1115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:38031 every 8 connection(s)
I20251029 08:52:01.584002 980 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/data/info.pb
I20251029 08:52:01.589627 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 980
I20251029 08:52:01.589823 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/instance
I20251029 08:52:01.622272 1118 heartbeater.cc:344] Connected to a master server at 127.0.104.61:38825
I20251029 08:52:01.622777 1118 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:01.623986 1118 heartbeater.cc:507] Master 127.0.104.61:38825 requested a full tablet report, sending...
I20251029 08:52:01.625475 1116 heartbeater.cc:344] Connected to a master server at 127.0.104.60:42135
I20251029 08:52:01.625607 1116 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:01.626041 1116 heartbeater.cc:507] Master 127.0.104.60:42135 requested a full tablet report, sending...
I20251029 08:52:01.626269 526 ts_manager.cc:194] Registered new tserver with Master: 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:01.626945 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:01.626935 597 ts_manager.cc:194] Registered new tserver with Master: 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:01.627076 1117 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:01.627601 1117 heartbeater.cc:507] Master 127.0.104.62:40435 requested a full tablet report, sending...
I20251029 08:52:01.627990 597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:60007
I20251029 08:52:01.628628 454 ts_manager.cc:194] Registered new tserver with Master: 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:01.640799 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251029 08:52:01.673678 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_051c9377bf26441383f3fa5a246c4ebf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.673717 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c52093d680ba4259ab6a46d4e84030ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.673784 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ccaea75df2cc4fdf8b163f9c828f04dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.675312 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_007e7b6e5ed74cd78121227c0b1fcc05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.675420 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bf8748f88d99499086cec85ebb81d806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.675741 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccaea75df2cc4fdf8b163f9c828f04dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.675661 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5dc52ed7835d4a92ab92d7f3aa6aa26f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.676074 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf8748f88d99499086cec85ebb81d806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.676362 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc52ed7835d4a92ab92d7f3aa6aa26f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.676335 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b856839bab3c4e65a7a7057b0411532d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.676891 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c52093d680ba4259ab6a46d4e84030ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.676867 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8ea0dafcdbb8433285d425bf20d14f0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.676930 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b856839bab3c4e65a7a7057b0411532d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.678179 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_051c9377bf26441383f3fa5a246c4ebf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.678488 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_007e7b6e5ed74cd78121227c0b1fcc05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.678495 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ea0dafcdbb8433285d425bf20d14f0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.684242 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:01.684548 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:01.684748 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d85bff1b12d64ff081b9c70738c1978f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.685645 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d85bff1b12d64ff081b9c70738c1978f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.719027 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_96176e6b1ffc4ff385b53df9e49cd0d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.720165 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96176e6b1ffc4ff385b53df9e49cd0d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.723184 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_051c9377bf26441383f3fa5a246c4ebf" } new_table_name: "default.Table_a09ec8042633426dada8c32d5b54974d" modify_external_catalogs: true
I20251029 08:52:01.726191 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c52093d680ba4259ab6a46d4e84030ff" } new_table_name: "default.Table_ec5e17a240a046d78a1c65dde9e90397" modify_external_catalogs: true
I20251029 08:52:01.729306 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bf8748f88d99499086cec85ebb81d806" } new_table_name: "default.Table_29397ee9f3114bbd8144fba923bab552" modify_external_catalogs: true
I20251029 08:52:01.736389 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5dc52ed7835d4a92ab92d7f3aa6aa26f" } new_table_name: "default.Table_97135f7d4b3749e797f32a7b46656e4f" modify_external_catalogs: true
I20251029 08:52:01.756837 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ccaea75df2cc4fdf8b163f9c828f04dc" } modify_external_catalogs: true
I20251029 08:52:01.757102 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ccaea75df2cc4fdf8b163f9c828f04dc" } modify_external_catalogs: true
I20251029 08:52:01.759701 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_007e7b6e5ed74cd78121227c0b1fcc05" } modify_external_catalogs: true
I20251029 08:52:01.759891 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_007e7b6e5ed74cd78121227c0b1fcc05" } modify_external_catalogs: true
I20251029 08:52:01.762552 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b856839bab3c4e65a7a7057b0411532d" } new_table_name: "default.Table_65df6e8062da4cadb06efa51355b2afb" modify_external_catalogs: true
W20251029 08:52:01.774904 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24826ee22d284341ac8ad0258ccfe567 consensus state has no leader
W20251029 08:52:01.776322 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15fe6b805d9d43889cff654f4641d989 consensus state has no leader
I20251029 08:52:01.777148 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ea0dafcdbb8433285d425bf20d14f0a" } modify_external_catalogs: true
I20251029 08:52:01.777386 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ea0dafcdbb8433285d425bf20d14f0a" } modify_external_catalogs: true
I20251029 08:52:01.777818 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_96176e6b1ffc4ff385b53df9e49cd0d4" } modify_external_catalogs: true
I20251029 08:52:01.778080 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_96176e6b1ffc4ff385b53df9e49cd0d4" } modify_external_catalogs: true
I20251029 08:52:01.780850 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec5e17a240a046d78a1c65dde9e90397" } modify_external_catalogs: true
I20251029 08:52:01.781167 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec5e17a240a046d78a1c65dde9e90397" } modify_external_catalogs: true
W20251029 08:52:01.782124 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f421e9fdfc482885de999ed9771256 consensus state has no leader
W20251029 08:52:01.796976 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ec8cb95194e4a7fb0002b6db8b9073a has no consensus state
I20251029 08:52:01.798365 590 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 28fcb0eef3f749b4a263cc81ec7a8d1e
I20251029 08:52:01.800205 593 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6a4571134a994d59a99581743cd6deee
I20251029 08:52:01.805478 597 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4330bd1ddca94516a159c9c970f005e4
W20251029 08:52:01.806979 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b17cb6f63a4064873d0851f4b563cb has no consensus state
I20251029 08:52:01.813324 764 tablet_service.cc:1505] Processing CreateTablet for tablet 24826ee22d284341ac8ad0258ccfe567 (DEFAULT_TABLE table=default.Table_c52093d680ba4259ab6a46d4e84030ff [id=2e26f7325f4640479006670b114d0f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.815140 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24826ee22d284341ac8ad0258ccfe567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.816010 592 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f051ff82eded47178511940b72f09acc
I20251029 08:52:01.822603 905 tablet_service.cc:1505] Processing CreateTablet for tablet 24826ee22d284341ac8ad0258ccfe567 (DEFAULT_TABLE table=default.Table_c52093d680ba4259ab6a46d4e84030ff [id=2e26f7325f4640479006670b114d0f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.824261 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24826ee22d284341ac8ad0258ccfe567. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:01.831542 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 24826ee22d284341ac8ad0258ccfe567 (table default.Table_ec5e17a240a046d78a1c65dde9e90397 [id=2e26f7325f4640479006670b114d0f59], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24826ee22d284341ac8ad0258ccfe567 consensus state has no leader
I20251029 08:52:01.821573 904 tablet_service.cc:1505] Processing CreateTablet for tablet 11f421e9fdfc482885de999ed9771256 (DEFAULT_TABLE table=default.Table_051c9377bf26441383f3fa5a246c4ebf [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.833837 763 tablet_service.cc:1505] Processing CreateTablet for tablet 11f421e9fdfc482885de999ed9771256 (DEFAULT_TABLE table=default.Table_051c9377bf26441383f3fa5a246c4ebf [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.834367 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f421e9fdfc482885de999ed9771256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.834702 903 tablet_service.cc:1505] Processing CreateTablet for tablet 15fe6b805d9d43889cff654f4641d989 (DEFAULT_TABLE table=default.Table_bf8748f88d99499086cec85ebb81d806 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.834002 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f421e9fdfc482885de999ed9771256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.835714 762 tablet_service.cc:1505] Processing CreateTablet for tablet 15fe6b805d9d43889cff654f4641d989 (DEFAULT_TABLE table=default.Table_bf8748f88d99499086cec85ebb81d806 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.836007 902 tablet_service.cc:1505] Processing CreateTablet for tablet 04b17cb6f63a4064873d0851f4b563cb (DEFAULT_TABLE table=default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.836201 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15fe6b805d9d43889cff654f4641d989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.836866 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b17cb6f63a4064873d0851f4b563cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.837373 761 tablet_service.cc:1505] Processing CreateTablet for tablet 04b17cb6f63a4064873d0851f4b563cb (DEFAULT_TABLE table=default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.837512 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15fe6b805d9d43889cff654f4641d989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.837783 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b17cb6f63a4064873d0851f4b563cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.838856 901 tablet_service.cc:1505] Processing CreateTablet for tablet ec0737021efd4489a1ee32c3fc760b0d (DEFAULT_TABLE table=default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.838955 760 tablet_service.cc:1505] Processing CreateTablet for tablet ec0737021efd4489a1ee32c3fc760b0d (DEFAULT_TABLE table=default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.839358 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0737021efd4489a1ee32c3fc760b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.839404 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0737021efd4489a1ee32c3fc760b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.839325 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:01.840385 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a (DEFAULT_TABLE table=default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.840384 759 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a (DEFAULT_TABLE table=default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.840854 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec8cb95194e4a7fb0002b6db8b9073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.841364 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec8cb95194e4a7fb0002b6db8b9073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.842501 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:01.844761 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:01.856354 971 heartbeater.cc:499] Master 127.0.104.60:42135 was elected leader, sending a full tablet report...
W20251029 08:52:01.856747 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 24826ee22d284341ac8ad0258ccfe567 because the tablet was not found. No further retry: Not found: Tablet not found: 24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:01.864197 1177 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:01.874617 829 heartbeater.cc:499] Master 127.0.104.60:42135 was elected leader, sending a full tablet report...
I20251029 08:52:01.877741 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 1 got 0
I20251029 08:52:01.879995 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 15fe6b805d9d43889cff654f4641d989 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:01.884364 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 15fe6b805d9d43889cff654f4641d989 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:01.890941 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 15fe6b805d9d43889cff654f4641d989
W20251029 08:52:01.891788 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 24826ee22d284341ac8ad0258ccfe567 because the tablet was not found. No further retry: Not found: Tablet not found: 24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:01.892156 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 24826ee22d284341ac8ad0258ccfe567 (DEFAULT_TABLE table=default.Table_c52093d680ba4259ab6a46d4e84030ff [id=2e26f7325f4640479006670b114d0f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.893057 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:01.894158 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24826ee22d284341ac8ad0258ccfe567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.900815 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 11f421e9fdfc482885de999ed9771256 (DEFAULT_TABLE table=default.Table_051c9377bf26441383f3fa5a246c4ebf [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.901417 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f421e9fdfc482885de999ed9771256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.905694 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 15fe6b805d9d43889cff654f4641d989 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:01.906972 591 catalog_manager.cc:6719] ReplaceTablet: tablet 15fe6b805d9d43889cff654f4641d989 deleted and replaced by tablet e9af06b4d79441968bda6bd7d67d76bf
W20251029 08:52:01.908526 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 15fe6b805d9d43889cff654f4641d989 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15fe6b805d9d43889cff654f4641d989 already in progress: creating tablet
I20251029 08:52:01.908720 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 15fe6b805d9d43889cff654f4641d989 (DEFAULT_TABLE table=default.Table_bf8748f88d99499086cec85ebb81d806 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.909242 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15fe6b805d9d43889cff654f4641d989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.909765 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:01.910249 904 tablet_service.cc:1505] Processing CreateTablet for tablet e9af06b4d79441968bda6bd7d67d76bf (DEFAULT_TABLE table=default.Table_29397ee9f3114bbd8144fba923bab552 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.911113 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9af06b4d79441968bda6bd7d67d76bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.916720 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 04b17cb6f63a4064873d0851f4b563cb (DEFAULT_TABLE table=default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.917169 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b17cb6f63a4064873d0851f4b563cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.918171 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_98a6626e61584e298b46fadecb445bf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.918735 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98a6626e61584e298b46fadecb445bf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.923939 1047 tablet_service.cc:1505] Processing CreateTablet for tablet ec0737021efd4489a1ee32c3fc760b0d (DEFAULT_TABLE table=default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.924242 1178 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:01.925015 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0737021efd4489a1ee32c3fc760b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.929396 1178 tablet_bootstrap.cc:654] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:01.929883 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d5fa14f58bfd4b33bc71630503d6e762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.930460 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5fa14f58bfd4b33bc71630503d6e762 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.930539 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:01.930989 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:01.931805 759 tablet_service.cc:1505] Processing CreateTablet for tablet e9af06b4d79441968bda6bd7d67d76bf (DEFAULT_TABLE table=default.Table_29397ee9f3114bbd8144fba923bab552 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.931881 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a (DEFAULT_TABLE table=default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.932242 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9af06b4d79441968bda6bd7d67d76bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.932430 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec8cb95194e4a7fb0002b6db8b9073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.935230 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_935e9f61644a475fa4158f5807ff839b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.935746 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_935e9f61644a475fa4158f5807ff839b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.935813 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 11f421e9fdfc482885de999ed9771256 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:01.936156 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:01.939563 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a73e6fbacb424a2693d350f298cace67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.940208 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a73e6fbacb424a2693d350f298cace67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.941843 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ec8cb95194e4a7fb0002b6db8b9073a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:01.944511 1178 log.cc:826] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Log is configured to *not* fsync() on all Append() calls
I20251029 08:52:01.945462 1045 tablet_service.cc:1505] Processing CreateTablet for tablet e9af06b4d79441968bda6bd7d67d76bf (DEFAULT_TABLE table=default.Table_29397ee9f3114bbd8144fba923bab552 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.946002 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9af06b4d79441968bda6bd7d67d76bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.946799 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1cb50d7e539c415cacec3985a05ea9b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.946849 905 tablet_service.cc:1505] Processing CreateTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 (DEFAULT_TABLE table=default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:01.947896 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cb50d7e539c415cacec3985a05ea9b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.949527 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.950150 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.951802 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1c1bec248bbd44408a382604c06f75f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:01.952318 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_20e14b5aff1e4ecc878dc58302b78794"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.952932 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20e14b5aff1e4ecc878dc58302b78794 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.953256 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 24826ee22d284341ac8ad0258ccfe567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24826ee22d284341ac8ad0258ccfe567 already in progress: creating tablet
W20251029 08:52:01.953555 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 15fe6b805d9d43889cff654f4641d989 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15fe6b805d9d43889cff654f4641d989 already in progress: creating tablet
W20251029 08:52:01.954901 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c1bec248bbd44408a382604c06f75f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:01.954941 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 (DEFAULT_TABLE table=default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.955487 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987fb4b242c4f2e80b0ec861999f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.955996 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987fb4b242c4f2e80b0ec861999f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.963191 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9bd45731144649fdb51ea314cb85ed13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:01.963793 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bd45731144649fdb51ea314cb85ed13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:01.970523 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 24826ee22d284341ac8ad0258ccfe567 because the tablet was not found. No further retry: Not found: Tablet not found: 24826ee22d284341ac8ad0258ccfe567
W20251029 08:52:01.970819 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 15fe6b805d9d43889cff654f4641d989 because the tablet was not found. No further retry: Not found: Tablet not found: 15fe6b805d9d43889cff654f4641d989
I20251029 08:52:01.973866 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9ec8cb95194e4a7fb0002b6db8b9073a
I20251029 08:52:01.974354 590 catalog_manager.cc:6719] ReplaceTablet: tablet 9ec8cb95194e4a7fb0002b6db8b9073a deleted and replaced by tablet 0c4d04b8441643728fcb8aa881157c50
I20251029 08:52:01.974704 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 11f421e9fdfc482885de999ed9771256
I20251029 08:52:01.975114 597 catalog_manager.cc:6719] ReplaceTablet: tablet 11f421e9fdfc482885de999ed9771256 deleted and replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd
I20251029 08:52:01.975971 1177 tablet_bootstrap.cc:654] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:01.976612 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:01.977609 759 tablet_service.cc:1505] Processing CreateTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 (DEFAULT_TABLE table=default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:01.977926 1178 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:01.978616 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:01.978711 1178 ts_tablet_manager.cc:1403] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.004s
I20251029 08:52:01.979727 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:01.980836 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:01.981536 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9ec8cb95194e4a7fb0002b6db8b9073a because the tablet was not found. No further retry: Not found: Tablet not found: 9ec8cb95194e4a7fb0002b6db8b9073a
W20251029 08:52:01.981863 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 11f421e9fdfc482885de999ed9771256 because the tablet was not found. No further retry: Not found: Tablet not found: 11f421e9fdfc482885de999ed9771256
W20251029 08:52:01.982120 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9ec8cb95194e4a7fb0002b6db8b9073a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ec8cb95194e4a7fb0002b6db8b9073a already in progress: creating tablet
W20251029 08:52:01.982424 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 11f421e9fdfc482885de999ed9771256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11f421e9fdfc482885de999ed9771256 already in progress: creating tablet
I20251029 08:52:01.983888 1183 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:01.978101 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987fb4b242c4f2e80b0ec861999f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:01.976413 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:01.978238 905 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:01.989717 1177 log.cc:826] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Log is configured to *not* fsync() on all Append() calls
W20251029 08:52:01.990793 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 11f421e9fdfc482885de999ed9771256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11f421e9fdfc482885de999ed9771256 already in progress: creating tablet
W20251029 08:52:01.991056 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9ec8cb95194e4a7fb0002b6db8b9073a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ec8cb95194e4a7fb0002b6db8b9073a already in progress: creating tablet
I20251029 08:52:01.994616 1183 tablet_bootstrap.cc:654] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.002326 1183 log.cc:826] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Log is configured to *not* fsync() on all Append() calls
I20251029 08:52:02.003926 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bfa1f140404f4768b6ae7c7fabc75d26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.004739 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.005872 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a09ec8042633426dada8c32d5b54974d" } new_table_name: "default.Table_5a03fa86d2bc462a92a5d3e289c6a669" modify_external_catalogs: true
I20251029 08:52:02.006322 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a73e6fbacb424a2693d350f298cace67" } new_table_name: "default.Table_bcb8471db41b49f7ae96f3818b371219" modify_external_catalogs: true
I20251029 08:52:02.008296 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_97135f7d4b3749e797f32a7b46656e4f" } modify_external_catalogs: true
I20251029 08:52:02.008499 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_97135f7d4b3749e797f32a7b46656e4f" } modify_external_catalogs: true
I20251029 08:52:02.010165 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d5fa14f58bfd4b33bc71630503d6e762" } new_table_name: "default.Table_e74a30852a7a4e948715aec58d12aa94" modify_external_catalogs: true
I20251029 08:52:02.011771 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_65df6e8062da4cadb06efa51355b2afb" } new_table_name: "default.Table_9e4d388a3efe41e7b3c28c970323c49d" modify_external_catalogs: true
I20251029 08:52:02.013161 1183 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.014155 1183 ts_tablet_manager.cc:1403] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20251029 08:52:02.017890 1178 raft_consensus.cc:359] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.023516 1178 raft_consensus.cc:385] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.023757 1178 raft_consensus.cc:740] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.024955 1178 consensus_queue.cc:260] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.026546 1183 raft_consensus.cc:359] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.027086 1183 raft_consensus.cc:385] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.027230 1183 raft_consensus.cc:740] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.027959 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu pid 424
I20251029 08:52:02.029610 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 08:52:02.031957 1178 ts_tablet_manager.cc:1434] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.053s user 0.009s sys 0.007s
I20251029 08:52:02.034914 1177 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.035285 1177 ts_tablet_manager.cc:1403] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.171s user 0.011s sys 0.015s
I20251029 08:52:02.042951 1178 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.042891 1177 raft_consensus.cc:359] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.043579 1177 raft_consensus.cc:385] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.043519 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98a6626e61584e298b46fadecb445bf0" } modify_external_catalogs: true
I20251029 08:52:02.043737 1177 raft_consensus.cc:740] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.043771 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98a6626e61584e298b46fadecb445bf0" } modify_external_catalogs: true
I20251029 08:52:02.043076 1183 consensus_queue.cc:260] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.044632 1177 consensus_queue.cc:260] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.046917 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 62e136716a2344ee8a7ada59e397bb92 (DEFAULT_TABLE table=default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.046945 759 tablet_service.cc:1505] Processing CreateTablet for tablet 62e136716a2344ee8a7ada59e397bb92 (DEFAULT_TABLE table=default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.047418 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e136716a2344ee8a7ada59e397bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.048403 761 tablet_service.cc:1505] Processing CreateTablet for tablet 49cb49df9bee421188997cde00967852 (DEFAULT_TABLE table=default.Table_d5fa14f58bfd4b33bc71630503d6e762 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.049353 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_29397ee9f3114bbd8144fba923bab552" } new_table_name: "default.Table_e0794b7ebfe242f3b2dc881492a95de6" modify_external_catalogs: true
I20251029 08:52:02.049540 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d85bff1b12d64ff081b9c70738c1978f" } modify_external_catalogs: true
I20251029 08:52:02.049762 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d85bff1b12d64ff081b9c70738c1978f" } modify_external_catalogs: true
I20251029 08:52:02.050781 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_935e9f61644a475fa4158f5807ff839b" } modify_external_catalogs: true
I20251029 08:52:02.050957 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_935e9f61644a475fa4158f5807ff839b" } modify_external_catalogs: true
I20251029 08:52:02.051515 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e136716a2344ee8a7ada59e397bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.051502 905 tablet_service.cc:1505] Processing CreateTablet for tablet 62e136716a2344ee8a7ada59e397bb92 (DEFAULT_TABLE table=default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.051921 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e136716a2344ee8a7ada59e397bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.052239 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb49df9bee421188997cde00967852. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.056361 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 728e64abaadc46598bf2dbc32dc55bdd has no consensus state
I20251029 08:52:02.056437 902 tablet_service.cc:1505] Processing CreateTablet for tablet 49cb49df9bee421188997cde00967852 (DEFAULT_TABLE table=default.Table_d5fa14f58bfd4b33bc71630503d6e762 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.056840 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb49df9bee421188997cde00967852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.059477 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9bd45731144649fdb51ea314cb85ed13" } modify_external_catalogs: true
I20251029 08:52:02.059687 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9bd45731144649fdb51ea314cb85ed13" } modify_external_catalogs: true
W20251029 08:52:02.067718 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7f8ae2eeabd419b8999372e6aa922ae has no consensus state
I20251029 08:52:02.069242 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 49cb49df9bee421188997cde00967852 (DEFAULT_TABLE table=default.Table_d5fa14f58bfd4b33bc71630503d6e762 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.069835 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb49df9bee421188997cde00967852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.076104 1177 ts_tablet_manager.cc:1434] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.041s user 0.011s sys 0.004s
I20251029 08:52:02.078327 1178 tablet_bootstrap.cc:654] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.078938 594 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 0c4d04b8441643728fcb8aa881157c50
W20251029 08:52:02.079875 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49cb49df9bee421188997cde00967852 consensus state has no leader
I20251029 08:52:02.080391 1177 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:02.082480 974 tablet.cc:2378] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:02.082796 1177 tablet_bootstrap.cc:654] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.093396 1178 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.093693 1178 ts_tablet_manager.cc:1403] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.002s
I20251029 08:52:02.094728 1178 raft_consensus.cc:359] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.094986 1178 raft_consensus.cc:385] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.095114 1178 raft_consensus.cc:740] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.095407 1178 consensus_queue.cc:260] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.097760 1178 ts_tablet_manager.cc:1434] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:02.097729 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7987fb4b242c4f2e80b0ec861999f405
I20251029 08:52:02.098795 1178 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.099651 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.099735 1116 heartbeater.cc:499] Master 127.0.104.60:42135 was elected leader, sending a full tablet report...
W20251029 08:52:02.100551 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7987fb4b242c4f2e80b0ec861999f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7987fb4b242c4f2e80b0ec861999f405 already in progress: creating tablet
I20251029 08:52:02.100961 905 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.104203 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7987fb4b242c4f2e80b0ec861999f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7987fb4b242c4f2e80b0ec861999f405 already in progress: creating tablet
I20251029 08:52:02.099388 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.106006 1183 ts_tablet_manager.cc:1434] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.091s user 0.010s sys 0.025s
W20251029 08:52:02.106007 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7987fb4b242c4f2e80b0ec861999f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7987fb4b242c4f2e80b0ec861999f405 already in progress: creating tablet
I20251029 08:52:02.106494 1183 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.108839 1183 tablet_bootstrap.cc:654] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.109283 1177 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
W20251029 08:52:02.109236 832 tablet.cc:2378] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:02.109508 1177 ts_tablet_manager.cc:1403] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.017s
I20251029 08:52:02.109681 1178 tablet_bootstrap.cc:654] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.110620 1177 raft_consensus.cc:359] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.110910 1177 raft_consensus.cc:385] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.111058 1177 raft_consensus.cc:740] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.111402 1177 consensus_queue.cc:260] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.112131 1177 ts_tablet_manager.cc:1434] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:02.112464 1177 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.113296 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 49cb49df9bee421188997cde00967852 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.114210 1177 tablet_bootstrap.cc:654] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.115218 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 11f421e9fdfc482885de999ed9771256 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.121248 1178 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.121248 1183 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.121408 1183 ts_tablet_manager.cc:1403] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.002s
I20251029 08:52:02.121425 1178 ts_tablet_manager.cc:1403] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20251029 08:52:02.122169 1178 raft_consensus.cc:359] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.122372 1178 raft_consensus.cc:385] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.122476 1178 raft_consensus.cc:740] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.122387 1183 raft_consensus.cc:359] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.122570 1183 raft_consensus.cc:385] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.122666 1183 raft_consensus.cc:740] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.122781 1178 consensus_queue.cc:260] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.122917 1183 consensus_queue.cc:260] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.123399 1183 ts_tablet_manager.cc:1434] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.124693 1178 ts_tablet_manager.cc:1434] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:02.124745 1183 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.124990 1178 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.126775 1183 tablet_bootstrap.cc:654] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.130666 1178 tablet_bootstrap.cc:654] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.133545 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ec8cb95194e4a7fb0002b6db8b9073a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.134187 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a398250eedaa4fd1bd18ef32c253a954"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.134941 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a398250eedaa4fd1bd18ef32c253a954 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.136451 1178 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.136585 1178 ts_tablet_manager.cc:1403] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20251029 08:52:02.136695 1193 raft_consensus.cc:493] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.136941 1193 raft_consensus.cc:515] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.137821 1178 raft_consensus.cc:359] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.137995 1178 raft_consensus.cc:385] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.138065 1178 raft_consensus.cc:740] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.138290 1178 consensus_queue.cc:260] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.138971 590 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 79531a5e85ae41aebfe570e956031810
I20251029 08:52:02.139108 1193 leader_election.cc:290] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.140911 1178 ts_tablet_manager.cc:1434] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 08:52:02.141068 1183 tablet_bootstrap.cc:492] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.141252 1183 ts_tablet_manager.cc:1403] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251029 08:52:02.142051 1183 raft_consensus.cc:359] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.142267 1183 raft_consensus.cc:385] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.142349 1183 raft_consensus.cc:740] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.142621 1183 consensus_queue.cc:260] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.143136 1183 ts_tablet_manager.cc:1434] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:02.143568 1183 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.143606 1178 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.145305 1178 tablet_bootstrap.cc:654] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.145305 1183 tablet_bootstrap.cc:654] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.149982 1178 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.150143 1178 ts_tablet_manager.cc:1403] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251029 08:52:02.151213 1190 raft_consensus.cc:493] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.151414 1190 raft_consensus.cc:515] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.153424 1190 leader_election.cc:290] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.159041 597 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 1 got 0
I20251029 08:52:02.161559 1183 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.161767 1183 ts_tablet_manager.cc:1403] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20251029 08:52:02.162533 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_65df6e8062da4cadb06efa51355b2afb [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 1 got 0
I20251029 08:52:02.162781 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_29397ee9f3114bbd8144fba923bab552 [id=407839d19657473883ad497873b78fb2]). Expected version 1 got 0
I20251029 08:52:02.162854 1183 raft_consensus.cc:359] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.163026 1183 raft_consensus.cc:385] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.163129 1183 raft_consensus.cc:740] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.163386 1183 consensus_queue.cc:260] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:02.163585 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b17cb6f63a4064873d0851f4b563cb consensus state has no leader
I20251029 08:52:02.163916 1183 ts_tablet_manager.cc:1434] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:02.164207 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.164258 1183 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:02.164996 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9af06b4d79441968bda6bd7d67d76bf consensus state has no leader
I20251029 08:52:02.165294 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 62e136716a2344ee8a7ada59e397bb92
I20251029 08:52:02.166021 1183 tablet_bootstrap.cc:654] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.166954 759 tablet_service.cc:1552] Processing DeleteTablet for tablet ec0737021efd4489a1ee32c3fc760b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.167372 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.167665 905 tablet_service.cc:1552] Processing DeleteTablet for tablet ec0737021efd4489a1ee32c3fc760b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.167930 1176 tablet_replica.cc:333] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.167927 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.168139 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.168735 1175 tablet_replica.cc:333] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.168776 1176 raft_consensus.cc:2243] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.150822 1178 raft_consensus.cc:359] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.168921 1178 raft_consensus.cc:385] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.169001 1178 raft_consensus.cc:740] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.169152 1175 raft_consensus.cc:2243] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.169193 1176 raft_consensus.cc:2272] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.169503 1175 raft_consensus.cc:2272] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.169600 1178 consensus_queue.cc:260] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.171350 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.171789 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet ec0737021efd4489a1ee32c3fc760b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.171927 1176 ts_tablet_manager.cc:1916] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.172061 1179 tablet_replica.cc:333] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.172475 1179 raft_consensus.cc:2243] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.172784 1179 raft_consensus.cc:2272] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.173815 1178 ts_tablet_manager.cc:1434] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251029 08:52:02.174539 1179 ts_tablet_manager.cc:1916] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:02.174669 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 62e136716a2344ee8a7ada59e397bb92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e136716a2344ee8a7ada59e397bb92 already in progress: creating tablet
I20251029 08:52:02.177187 1178 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.177662 1175 ts_tablet_manager.cc:1916] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.178931 1178 tablet_bootstrap.cc:654] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.180603 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.181419 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 11f421e9fdfc482885de999ed9771256 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.185436 589 master_service.cc:954] ReplaceTablet: received request to replace tablet ec0737021efd4489a1ee32c3fc760b0d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.187748 1176 ts_tablet_manager.cc:1929] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.187989 1176 log.cc:1199] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.190060 1175 ts_tablet_manager.cc:1929] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.190339 1175 log.cc:1199] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.190855 1175 ts_tablet_manager.cc:1950] T ec0737021efd4489a1ee32c3fc760b0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.192778 1175 tablet_replica.cc:333] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.192962 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.194137 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.194260 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ec0737021efd4489a1ee32c3fc760b0d (table default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]) successfully deleted
I20251029 08:52:02.194337 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.194617 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.195042 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.196044 1051 tablet_service.cc:1505] Processing CreateTablet for tablet aa73cfd89a724aada8af426a56b3481a (DEFAULT_TABLE table=default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.196478 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa73cfd89a724aada8af426a56b3481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.200593 1179 ts_tablet_manager.cc:1929] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.200860 1179 log.cc:1199] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.201416 1179 ts_tablet_manager.cc:1950] T ec0737021efd4489a1ee32c3fc760b0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:02.203265 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ec0737021efd4489a1ee32c3fc760b0d (table default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]) successfully deleted
W20251029 08:52:02.203548 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 62e136716a2344ee8a7ada59e397bb92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e136716a2344ee8a7ada59e397bb92 already in progress: creating tablet
W20251029 08:52:02.203923 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 24826ee22d284341ac8ad0258ccfe567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24826ee22d284341ac8ad0258ccfe567 already in progress: creating tablet
I20251029 08:52:02.204632 1179 tablet_replica.cc:333] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.204881 1179 raft_consensus.cc:2243] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:02.204993 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9ec8cb95194e4a7fb0002b6db8b9073a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ec8cb95194e4a7fb0002b6db8b9073a already in progress: creating tablet
I20251029 08:52:02.205176 1179 raft_consensus.cc:2272] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.206102 1046 tablet_service.cc:1505] Processing CreateTablet for tablet de8f868095a642f6a5d70c30bb74105c (DEFAULT_TABLE table=default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8 [id=520b9919efc542c095d65a47ce1823ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.206501 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8f868095a642f6a5d70c30bb74105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.211982 1179 ts_tablet_manager.cc:1916] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.212630 1045 tablet_service.cc:1505] Processing CreateTablet for tablet b4a3272a7a0a419dadd2cad132e996fd (DEFAULT_TABLE table=default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.213001 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a3272a7a0a419dadd2cad132e996fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.191852 759 tablet_service.cc:1505] Processing CreateTablet for tablet aa73cfd89a724aada8af426a56b3481a (DEFAULT_TABLE table=default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.218549 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa73cfd89a724aada8af426a56b3481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.220129 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_68f6f1f513884300822e1412378a89b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.220778 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68f6f1f513884300822e1412378a89b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.221246 597 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_9e4d388a3efe41e7b3c28c970323c49d [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 2 got 0
I20251029 08:52:02.221835 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_35f72924484241fbaf61c2cafa158f4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.222390 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35f72924484241fbaf61c2cafa158f4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.222452 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.222695 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d65e71e019114f57877994e1b43540ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.222934 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:02.223156 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d65e71e019114f57877994e1b43540ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.196097 1178 tablet_bootstrap.cc:492] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.224211 1178 ts_tablet_manager.cc:1403] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20251029 08:52:02.225174 1178 raft_consensus.cc:359] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.225358 1178 raft_consensus.cc:385] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.225459 1178 raft_consensus.cc:740] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.225682 1178 consensus_queue.cc:260] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.226205 1178 ts_tablet_manager.cc:1434] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:02.226506 1178 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.201949 761 tablet_service.cc:1505] Processing CreateTablet for tablet de8f868095a642f6a5d70c30bb74105c (DEFAULT_TABLE table=default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8 [id=520b9919efc542c095d65a47ce1823ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.229040 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8f868095a642f6a5d70c30bb74105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.229152 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_9e4d388a3efe41e7b3c28c970323c49d [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 2 got 0
I20251029 08:52:02.229333 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]). Expected version 2 got 0
W20251029 08:52:02.202587 718 leader_election.cc:343] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.233702 764 tablet_service.cc:1505] Processing CreateTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae (DEFAULT_TABLE table=default.Table_bcb8471db41b49f7ae96f3818b371219 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.233949 762 tablet_service.cc:1505] Processing CreateTablet for tablet b4a3272a7a0a419dadd2cad132e996fd (DEFAULT_TABLE table=default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.234071 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f8ae2eeabd419b8999372e6aa922ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.234333 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a3272a7a0a419dadd2cad132e996fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.236304 763 tablet_service.cc:1505] Processing CreateTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd (DEFAULT_TABLE table=default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.236697 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e64abaadc46598bf2dbc32dc55bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.235618 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0ee7a5a203a24a049f5a8ef903dd6aff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.238835 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ee7a5a203a24a049f5a8ef903dd6aff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.240273 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d881df3a5f1043079e80ac59d710e3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.240820 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d881df3a5f1043079e80ac59d710e3ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.242251 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.242682 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.243889 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a2cb1179d9de4d818e95e90c74afef35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.244347 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2cb1179d9de4d818e95e90c74afef35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.246778 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.246910 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.247081 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 15fe6b805d9d43889cff654f4641d989 with delete_type TABLET_DATA_DELETED (replaced by tablet e9af06b4d79441968bda6bd7d67d76bf) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.247144 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.251765 1178 tablet_bootstrap.cc:654] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.252503 758 tablet_service.cc:1505] Processing CreateTablet for tablet 710e99b4116744289f9a2130f65f4b94 (DEFAULT_TABLE table=default.Table_1cb50d7e539c415cacec3985a05ea9b6 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.252966 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e99b4116744289f9a2130f65f4b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.267313 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_087d939fe42c43c8baef8a273f08eda5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.268313 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_087d939fe42c43c8baef8a273f08eda5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.269534 1176 ts_tablet_manager.cc:1950] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.270567 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ec0737021efd4489a1ee32c3fc760b0d (table default.Table_d85bff1b12d64ff081b9c70738c1978f [id=2f81fea48ac841e389f822a9c2a3b1de]) successfully deleted
I20251029 08:52:02.271286 1179 ts_tablet_manager.cc:1929] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:02.272244 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 62e136716a2344ee8a7ada59e397bb92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e136716a2344ee8a7ada59e397bb92 already in progress: creating tablet
I20251029 08:52:02.272563 1176 tablet_replica.cc:333] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.272774 1176 raft_consensus.cc:2243] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.272835 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd (DEFAULT_TABLE table=default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.272951 1176 raft_consensus.cc:2272] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.272987 1051 tablet_service.cc:1505] Processing CreateTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae (DEFAULT_TABLE table=default.Table_bcb8471db41b49f7ae96f3818b371219 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.273396 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e64abaadc46598bf2dbc32dc55bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.273620 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f8ae2eeabd419b8999372e6aa922ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.273773 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet ec0737021efd4489a1ee32c3fc760b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.273837 1176 ts_tablet_manager.cc:1916] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.273967 1178 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.273978 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 710e99b4116744289f9a2130f65f4b94 (DEFAULT_TABLE table=default.Table_1cb50d7e539c415cacec3985a05ea9b6 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.274135 1178 ts_tablet_manager.cc:1403] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.006s
I20251029 08:52:02.274294 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e99b4116744289f9a2130f65f4b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.275506 1178 raft_consensus.cc:359] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.275743 1178 raft_consensus.cc:385] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.275840 1178 raft_consensus.cc:740] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.276163 1178 consensus_queue.cc:260] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.278116 1178 ts_tablet_manager.cc:1434] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251029 08:52:02.278463 1178 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.281293 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 49cb49df9bee421188997cde00967852
I20251029 08:52:02.281795 588 catalog_manager.cc:6719] ReplaceTablet: tablet 49cb49df9bee421188997cde00967852 deleted and replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9
I20251029 08:52:02.282894 1178 tablet_bootstrap.cc:654] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.283860 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.284055 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.284286 1179 log.cc:1199] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/15fe6b805d9d43889cff654f4641d989
I20251029 08:52:02.284782 1179 ts_tablet_manager.cc:1950] T 15fe6b805d9d43889cff654f4641d989 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:02.285954 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 15fe6b805d9d43889cff654f4641d989 (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:02.288218 1176 ts_tablet_manager.cc:1929] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:02.288614 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7987fb4b242c4f2e80b0ec861999f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7987fb4b242c4f2e80b0ec861999f405 already in progress: creating tablet
W20251029 08:52:02.288945 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 11f421e9fdfc482885de999ed9771256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11f421e9fdfc482885de999ed9771256 already in progress: creating tablet
I20251029 08:52:02.288995 1176 log.cc:1199] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:02.289609 1176 ts_tablet_manager.cc:1950] T 24826ee22d284341ac8ad0258ccfe567 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:02.290421 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ec0737021efd4489a1ee32c3fc760b0d because the tablet was not found. No further retry: Not found: Tablet not found: ec0737021efd4489a1ee32c3fc760b0d
W20251029 08:52:02.290736 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 49cb49df9bee421188997cde00967852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49cb49df9bee421188997cde00967852 already in progress: creating tablet
I20251029 08:52:02.292078 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 24826ee22d284341ac8ad0258ccfe567 (table default.Table_ec5e17a240a046d78a1c65dde9e90397 [id=2e26f7325f4640479006670b114d0f59]) successfully deleted
I20251029 08:52:02.292693 1176 tablet_replica.cc:333] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.293339 1183 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.293576 1183 ts_tablet_manager.cc:1403] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.129s user 0.032s sys 0.044s
I20251029 08:52:02.294931 1183 raft_consensus.cc:359] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.295229 1183 raft_consensus.cc:385] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.295364 1183 raft_consensus.cc:740] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.295854 1183 consensus_queue.cc:260] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.296500 1183 ts_tablet_manager.cc:1434] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:02.296837 1183 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.298517 1183 tablet_bootstrap.cc:654] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.298748 1176 raft_consensus.cc:2243] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.298979 1176 raft_consensus.cc:2272] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.299744 1176 ts_tablet_manager.cc:1916] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.193881 902 tablet_service.cc:1505] Processing CreateTablet for tablet aa73cfd89a724aada8af426a56b3481a (DEFAULT_TABLE table=default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.301589 1175 raft_consensus.cc:2243] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.301776 1175 raft_consensus.cc:2272] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.301925 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa73cfd89a724aada8af426a56b3481a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.204744 904 tablet_service.cc:1505] Processing CreateTablet for tablet de8f868095a642f6a5d70c30bb74105c (DEFAULT_TABLE table=default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8 [id=520b9919efc542c095d65a47ce1823ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.206857 905 tablet_service.cc:1505] Processing CreateTablet for tablet b4a3272a7a0a419dadd2cad132e996fd (DEFAULT_TABLE table=default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.209390 903 tablet_service.cc:1505] Processing CreateTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd (DEFAULT_TABLE table=default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.221199 900 tablet_service.cc:1505] Processing CreateTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae (DEFAULT_TABLE table=default.Table_bcb8471db41b49f7ae96f3818b371219 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.238379 901 tablet_service.cc:1552] Processing DeleteTablet for tablet ec0737021efd4489a1ee32c3fc760b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.253259 899 tablet_service.cc:1505] Processing CreateTablet for tablet 710e99b4116744289f9a2130f65f4b94 (DEFAULT_TABLE table=default.Table_1cb50d7e539c415cacec3985a05ea9b6 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.283159 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.307699 1176 ts_tablet_manager.cc:1929] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:02.310168 720 leader_election.cc:343] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: ec0737021efd4489a1ee32c3fc760b0d
I20251029 08:52:02.310308 1176 log.cc:1199] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9ec8cb95194e4a7fb0002b6db8b9073a
I20251029 08:52:02.310364 720 leader_election.cc:304] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.310693 1176 ts_tablet_manager.cc:1950] T 9ec8cb95194e4a7fb0002b6db8b9073a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:02.310853 720 raft_consensus.cc:2686] T ec0737021efd4489a1ee32c3fc760b0d P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:02.311709 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9ec8cb95194e4a7fb0002b6db8b9073a (table default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]) successfully deleted
I20251029 08:52:02.312551 1183 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.312700 1183 ts_tablet_manager.cc:1403] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20251029 08:52:02.313529 1183 raft_consensus.cc:359] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.313730 1183 raft_consensus.cc:385] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.313807 1183 raft_consensus.cc:740] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.314021 1183 consensus_queue.cc:260] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.315490 1183 ts_tablet_manager.cc:1434] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:02.315781 1183 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.316808 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8f868095a642f6a5d70c30bb74105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.317062 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a3272a7a0a419dadd2cad132e996fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.317278 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e99b4116744289f9a2130f65f4b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.318156 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f8ae2eeabd419b8999372e6aa922ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.319339 1183 tablet_bootstrap.cc:654] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.320096 1176 tablet_replica.cc:333] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.320231 1190 raft_consensus.cc:493] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.320290 1176 raft_consensus.cc:2243] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.320459 1176 raft_consensus.cc:2272] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.320420 1190 raft_consensus.cc:515] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.321071 1190 leader_election.cc:290] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.321460 1176 ts_tablet_manager.cc:1916] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.321758 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9af06b4d79441968bda6bd7d67d76bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:02.327236 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e64abaadc46598bf2dbc32dc55bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.327997 1176 ts_tablet_manager.cc:1929] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:02.331818 718 leader_election.cc:343] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.332286 1176 log.cc:1199] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/15fe6b805d9d43889cff654f4641d989
I20251029 08:52:02.332845 1176 ts_tablet_manager.cc:1950] T 15fe6b805d9d43889cff654f4641d989 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.334375 1176 tablet_replica.cc:333] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.334520 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 15fe6b805d9d43889cff654f4641d989 (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:02.335542 1176 raft_consensus.cc:2243] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.335491 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9af06b4d79441968bda6bd7d67d76bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:02.335943 1071 raft_consensus.cc:2468] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:02.336097 1176 raft_consensus.cc:2272] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.336900 720 leader_election.cc:304] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.337204 1176 ts_tablet_manager.cc:1916] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.337595 1190 raft_consensus.cc:2804] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:02.337751 1175 ts_tablet_manager.cc:1916] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.337755 1190 raft_consensus.cc:493] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251029 08:52:02.338019 1119 tablet.cc:2378] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:02.340113 1190 raft_consensus.cc:3060] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.349244 1190 raft_consensus.cc:515] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.352180 1190 leader_election.cc:290] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.352129 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9af06b4d79441968bda6bd7d67d76bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:02.352218 1176 ts_tablet_manager.cc:1929] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:02.353428 718 leader_election.cc:343] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.353595 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9af06b4d79441968bda6bd7d67d76bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:02.353807 1176 log.cc:1199] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/11f421e9fdfc482885de999ed9771256
I20251029 08:52:02.353801 1071 raft_consensus.cc:3060] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.354183 1176 ts_tablet_manager.cc:1950] T 11f421e9fdfc482885de999ed9771256 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.355149 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 11f421e9fdfc482885de999ed9771256 (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
W20251029 08:52:02.357014 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 49cb49df9bee421188997cde00967852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49cb49df9bee421188997cde00967852 already in progress: creating tablet
I20251029 08:52:02.357260 1071 raft_consensus.cc:2468] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:02.357715 720 leader_election.cc:304] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:02.358253 861 leader_election.cc:343] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 15fe6b805d9d43889cff654f4641d989
I20251029 08:52:02.358548 1190 raft_consensus.cc:2804] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:02.358814 861 leader_election.cc:343] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 15fe6b805d9d43889cff654f4641d989
I20251029 08:52:02.359174 861 leader_election.cc:304] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:02.359560 861 raft_consensus.cc:2686] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:02.362458 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1cb50d7e539c415cacec3985a05ea9b6" } new_table_name: "default.Table_1c778ace3a094ba7a37d7def4c6497a7" modify_external_catalogs: true
I20251029 08:52:02.365415 1175 ts_tablet_manager.cc:1929] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.365664 1190 raft_consensus.cc:697] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:02.365904 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.366261 1179 tablet_replica.cc:333] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.366245 1190 consensus_queue.cc:237] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.366492 1179 raft_consensus.cc:2243] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.366734 1179 raft_consensus.cc:2272] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.367828 1179 ts_tablet_manager.cc:1916] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.373611 1175 log.cc:1199] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/15fe6b805d9d43889cff654f4641d989
I20251029 08:52:02.374023 1175 ts_tablet_manager.cc:1950] T 15fe6b805d9d43889cff654f4641d989 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.375973 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 15fe6b805d9d43889cff654f4641d989 (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]) successfully deleted
W20251029 08:52:02.376312 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 62e136716a2344ee8a7ada59e397bb92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e136716a2344ee8a7ada59e397bb92 already in progress: creating tablet
W20251029 08:52:02.376571 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ec0737021efd4489a1ee32c3fc760b0d because the tablet was not found. No further retry: Not found: Tablet not found: ec0737021efd4489a1ee32c3fc760b0d
W20251029 08:52:02.376801 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 49cb49df9bee421188997cde00967852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49cb49df9bee421188997cde00967852 already in progress: creating tablet
I20251029 08:52:02.379735 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bcb8471db41b49f7ae96f3818b371219" } new_table_name: "default.Table_ffe5e7903bd1497391ecdeca67dae602" modify_external_catalogs: true
I20251029 08:52:02.381319 1179 ts_tablet_manager.cc:1929] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.381477 1179 log.cc:1199] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7987fb4b242c4f2e80b0ec861999f405
I20251029 08:52:02.382040 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.388855 1179 ts_tablet_manager.cc:1950] T 7987fb4b242c4f2e80b0ec861999f405 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:02.390444 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7987fb4b242c4f2e80b0ec861999f405 (table default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]) successfully deleted
I20251029 08:52:02.391103 1179 tablet_replica.cc:333] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.391434 1179 raft_consensus.cc:2243] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.391727 1179 raft_consensus.cc:2272] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.393218 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_20e14b5aff1e4ecc878dc58302b78794" } modify_external_catalogs: true
I20251029 08:52:02.393445 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_20e14b5aff1e4ecc878dc58302b78794" } modify_external_catalogs: true
I20251029 08:52:02.394141 1179 ts_tablet_manager.cc:1916] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.417163 1178 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.417363 1178 ts_tablet_manager.cc:1403] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.017s
I20251029 08:52:02.418532 1178 raft_consensus.cc:359] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.418756 1178 raft_consensus.cc:385] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.418846 1178 raft_consensus.cc:740] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.419996 1178 consensus_queue.cc:260] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.420574 1178 ts_tablet_manager.cc:1434] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251029 08:52:02.420887 1178 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.422338 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9e4d388a3efe41e7b3c28c970323c49d" } new_table_name: "default.Table_0377c0d70fc04e3b997daf88097cb74b" modify_external_catalogs: true
I20251029 08:52:02.423928 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c1bec248bbd44408a382604c06f75f1" } modify_external_catalogs: true
I20251029 08:52:02.424188 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c1bec248bbd44408a382604c06f75f1" } modify_external_catalogs: true
I20251029 08:52:02.424880 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 710e99b4116744289f9a2130f65f4b94 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.425196 1178 tablet_bootstrap.cc:654] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.428930 1177 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.429172 1177 ts_tablet_manager.cc:1403] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.317s user 0.004s sys 0.042s
I20251029 08:52:02.430209 1177 raft_consensus.cc:359] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.430503 1177 raft_consensus.cc:385] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.430588 1177 raft_consensus.cc:740] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.431115 1177 consensus_queue.cc:260] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.431736 1177 ts_tablet_manager.cc:1434] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251029 08:52:02.432103 1177 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.434653 1177 tablet_bootstrap.cc:654] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.434770 1178 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.434926 1178 ts_tablet_manager.cc:1403] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20251029 08:52:02.435717 1178 raft_consensus.cc:359] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.435909 1178 raft_consensus.cc:385] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.435967 1178 raft_consensus.cc:740] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.436216 1178 consensus_queue.cc:260] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.437189 1178 ts_tablet_manager.cc:1434] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:02.437484 1178 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.438990 1178 tablet_bootstrap.cc:654] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.453683 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 04b17cb6f63a4064873d0851f4b563cb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.455265 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.455615 1175 tablet_replica.cc:333] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.455806 1175 raft_consensus.cc:2243] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.455987 1175 raft_consensus.cc:2272] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.456754 1175 ts_tablet_manager.cc:1916] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.462945 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a398250eedaa4fd1bd18ef32c253a954" } modify_external_catalogs: true
I20251029 08:52:02.463176 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a398250eedaa4fd1bd18ef32c253a954" } modify_external_catalogs: true
I20251029 08:52:02.463788 1175 ts_tablet_manager.cc:1929] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.463981 1175 log.cc:1199] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9ec8cb95194e4a7fb0002b6db8b9073a
I20251029 08:52:02.464496 1175 ts_tablet_manager.cc:1950] T 9ec8cb95194e4a7fb0002b6db8b9073a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.465613 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9ec8cb95194e4a7fb0002b6db8b9073a (table default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]) successfully deleted
I20251029 08:52:02.468596 1190 raft_consensus.cc:493] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.468791 1190 raft_consensus.cc:515] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.469523 1190 leader_election.cc:290] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.470067 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b17cb6f63a4064873d0851f4b563cb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:02.470813 718 leader_election.cc:343] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.473506 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_87fd2963f55342f9a2a7c2dfda8f5fc8" } new_table_name: "default.Table_31cdca53404a4a36a5ee867f8aa53bfc" modify_external_catalogs: true
I20251029 08:52:02.474459 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu with pid 424
W20251029 08:52:02.481151 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 710e99b4116744289f9a2130f65f4b94 consensus state has no leader
I20251029 08:52:02.484264 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b17cb6f63a4064873d0851f4b563cb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:02.484536 1071 raft_consensus.cc:2468] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:02.484834 1183 tablet_bootstrap.cc:492] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.485001 1183 ts_tablet_manager.cc:1403] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.169s user 0.053s sys 0.086s
I20251029 08:52:02.485003 720 leader_election.cc:304] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.485522 1190 raft_consensus.cc:2804] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:02.485642 1190 raft_consensus.cc:493] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:02.485659 1177 tablet_bootstrap.cc:492] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.485729 1190 raft_consensus.cc:3060] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.485798 1177 ts_tablet_manager.cc:1403] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20251029 08:52:02.485795 1183 raft_consensus.cc:359] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.485994 1183 raft_consensus.cc:385] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.486064 1183 raft_consensus.cc:740] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.486289 1183 consensus_queue.cc:260] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.486569 1177 raft_consensus.cc:359] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.486756 1177 raft_consensus.cc:385] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.486836 1177 raft_consensus.cc:740] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.487059 1177 consensus_queue.cc:260] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.487579 1177 ts_tablet_manager.cc:1434] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.487818 1183 ts_tablet_manager.cc:1434] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20251029 08:52:02.488231 1183 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.488674 1177 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.490340 1183 tablet_bootstrap.cc:654] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.490343 1177 tablet_bootstrap.cc:654] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.491398 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_68f6f1f513884300822e1412378a89b0" } modify_external_catalogs: true
I20251029 08:52:02.491617 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_68f6f1f513884300822e1412378a89b0" } modify_external_catalogs: true
I20251029 08:52:02.491845 1190 raft_consensus.cc:515] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.493266 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b17cb6f63a4064873d0851f4b563cb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:02.493842 718 leader_election.cc:343] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.494673 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b17cb6f63a4064873d0851f4b563cb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:02.494865 1071 raft_consensus.cc:3060] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.495446 1190 leader_election.cc:290] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.498582 1179 ts_tablet_manager.cc:1929] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.498808 1179 log.cc:1199] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/11f421e9fdfc482885de999ed9771256
I20251029 08:52:02.499303 1179 ts_tablet_manager.cc:1950] T 11f421e9fdfc482885de999ed9771256 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:02.502255 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]). Expected version 2 got 0
I20251029 08:52:02.502290 1071 raft_consensus.cc:2468] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:02.502888 720 leader_election.cc:304] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.503257 1190 raft_consensus.cc:2804] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:02.503417 1190 raft_consensus.cc:697] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:02.503579 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 11f421e9fdfc482885de999ed9771256 (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
I20251029 08:52:02.503719 1190 consensus_queue.cc:237] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.504410 1183 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.504558 1183 ts_tablet_manager.cc:1403] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20251029 08:52:02.504765 761 tablet_service.cc:1505] Processing CreateTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.505203 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b8068c1c7ce49fc85b660f40a7d1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.505488 1183 raft_consensus.cc:359] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.505681 1183 raft_consensus.cc:385] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.505767 1183 raft_consensus.cc:740] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.506058 1183 consensus_queue.cc:260] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.506282 903 tablet_service.cc:1505] Processing CreateTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.506635 1183 ts_tablet_manager.cc:1434] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:02.506719 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b8068c1c7ce49fc85b660f40a7d1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.506902 1183 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.508663 1178 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.508791 1178 ts_tablet_manager.cc:1403] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.002s
I20251029 08:52:02.508888 1183 tablet_bootstrap.cc:654] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.509608 1178 raft_consensus.cc:359] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.509768 1178 raft_consensus.cc:385] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.509841 1178 raft_consensus.cc:740] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.510058 1178 consensus_queue.cc:260] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.510557 1178 ts_tablet_manager.cc:1434] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.510856 1178 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.516420 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.516903 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b8068c1c7ce49fc85b660f40a7d1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.518266 1178 tablet_bootstrap.cc:654] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:02.524796 997 connection.cc:537] client connection to 127.0.104.62:40435 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 08:52:02.525667 1117 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:40435 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 08:52:02.526000 830 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:40435 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.62:40435 (error 108)
I20251029 08:52:02.526391 1183 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.526340 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e74a30852a7a4e948715aec58d12aa94" } new_table_name: "default.Table_2520e1f6484548089ec0f601b0bd7cf9" modify_external_catalogs: true
I20251029 08:52:02.526595 1183 ts_tablet_manager.cc:1403] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20251029 08:52:02.527585 1183 raft_consensus.cc:359] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.527851 1183 raft_consensus.cc:385] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.527928 1183 raft_consensus.cc:740] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.528237 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec8cb95194e4a7fb0002b6db8b9073a with delete_type TABLET_DATA_DELETED (replaced by tablet 0c4d04b8441643728fcb8aa881157c50) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.528288 1183 consensus_queue.cc:260] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.528849 1183 ts_tablet_manager.cc:1434] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251029 08:52:02.529676 972 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:40435 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.62:40435 (error 108)
I20251029 08:52:02.530009 1183 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:02.530094 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:40435 (error 108)
I20251029 08:52:02.531778 1183 tablet_bootstrap.cc:654] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.532974 1179 tablet_replica.cc:333] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.533286 1179 raft_consensus.cc:2243] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.533825 1179 raft_consensus.cc:2272] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.535001 1179 ts_tablet_manager.cc:1916] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.535523 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.535835 1175 tablet_replica.cc:333] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.536060 1175 raft_consensus.cc:2243] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.536250 1175 raft_consensus.cc:2272] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.537895 1175 ts_tablet_manager.cc:1916] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.545226 1175 ts_tablet_manager.cc:1929] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.545403 1175 log.cc:1199] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/11f421e9fdfc482885de999ed9771256
I20251029 08:52:02.545783 1175 ts_tablet_manager.cc:1950] T 11f421e9fdfc482885de999ed9771256 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.546907 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 11f421e9fdfc482885de999ed9771256 (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
I20251029 08:52:02.551414 1178 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.551587 1178 ts_tablet_manager.cc:1403] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.005s
I20251029 08:52:02.552438 1177 tablet_bootstrap.cc:492] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.552603 1177 ts_tablet_manager.cc:1403] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.002s
I20251029 08:52:02.553519 1177 raft_consensus.cc:359] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.553715 1177 raft_consensus.cc:385] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.553788 1177 raft_consensus.cc:740] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.554042 1177 consensus_queue.cc:260] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.556332 1177 ts_tablet_manager.cc:1434] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:02.557401 1177 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.559003 1177 tablet_bootstrap.cc:654] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.560586 1178 raft_consensus.cc:359] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.560792 1178 raft_consensus.cc:385] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.560864 1178 raft_consensus.cc:740] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.561127 1178 consensus_queue.cc:260] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:02.562517 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7f8ae2eeabd419b8999372e6aa922ae consensus state has no leader
I20251029 08:52:02.562665 1178 ts_tablet_manager.cc:1434] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:02.563753 1178 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.565510 1178 tablet_bootstrap.cc:654] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.565999 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5a03fa86d2bc462a92a5d3e289c6a669" } modify_external_catalogs: true
I20251029 08:52:02.566207 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5a03fa86d2bc462a92a5d3e289c6a669" } modify_external_catalogs: true
I20251029 08:52:02.567291 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.567618 1177 tablet_bootstrap.cc:492] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.567761 1177 ts_tablet_manager.cc:1403] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251029 08:52:02.568539 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.568701 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f421e9fdfc482885de999ed9771256 with delete_type TABLET_DATA_DELETED (replaced by tablet 728e64abaadc46598bf2dbc32dc55bdd) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.569368 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 11f421e9fdfc482885de999ed9771256 because the tablet was not found. No further retry: Not found: Tablet not found: 11f421e9fdfc482885de999ed9771256
I20251029 08:52:02.569715 1177 raft_consensus.cc:359] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.569916 1177 raft_consensus.cc:385] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.570024 1177 raft_consensus.cc:740] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.570269 1177 consensus_queue.cc:260] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.572232 1177 ts_tablet_manager.cc:1434] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:02.573315 1177 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.575781 1177 tablet_bootstrap.cc:654] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.577982 1178 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.578123 1178 ts_tablet_manager.cc:1403] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251029 08:52:02.578867 1178 raft_consensus.cc:359] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.579058 1178 raft_consensus.cc:385] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.579141 1178 raft_consensus.cc:740] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.579973 1183 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.580152 1183 ts_tablet_manager.cc:1403] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.035s
W20251029 08:52:02.580232 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b17cb6f63a4064873d0851f4b563cb consensus state has no leader
I20251029 08:52:02.581033 593 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6fd4b705b45549bb8bc7ce4a1e328078
I20251029 08:52:02.581046 1183 raft_consensus.cc:359] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.581220 1183 raft_consensus.cc:385] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.581272 593 master_service.cc:954] ReplaceTablet: received request to replace tablet f7f8ae2eeabd419b8999372e6aa922ae from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.581321 1183 raft_consensus.cc:740] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.581585 1183 consensus_queue.cc:260] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.582336 1178 consensus_queue.cc:260] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:02.582597 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8f868095a642f6a5d70c30bb74105c consensus state has no leader
I20251029 08:52:02.582844 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ec8cb95194e4a7fb0002b6db8b9073a from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:02.583048 583 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b -> Peer f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435): Couldn't send request to peer f718bd0f00004f20a89d028a2fa8e6b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:02.584008 1178 ts_tablet_manager.cc:1434] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:02.584496 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a09dfe0f96c64010a72526b0f6cb3920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.584674 590 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 291723570eb844e5ac4e48b88a6d66a9
W20251029 08:52:02.585095 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a09dfe0f96c64010a72526b0f6cb3920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.585137 1178 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.582093 1183 ts_tablet_manager.cc:1434] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:02.585527 1183 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.586937 1178 tablet_bootstrap.cc:654] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.587740 1183 tablet_bootstrap.cc:654] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.588487 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 710e99b4116744289f9a2130f65f4b94
I20251029 08:52:02.588943 589 catalog_manager.cc:6719] ReplaceTablet: tablet 710e99b4116744289f9a2130f65f4b94 deleted and replaced by tablet b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:02.588934 1177 tablet_bootstrap.cc:492] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.589094 1177 ts_tablet_manager.cc:1403] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20251029 08:52:02.588960 1178 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.589689 1178 ts_tablet_manager.cc:1403] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251029 08:52:02.589936 1177 raft_consensus.cc:359] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.590122 1177 raft_consensus.cc:385] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.590189 1177 raft_consensus.cc:740] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.590461 1177 consensus_queue.cc:260] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.590663 1178 raft_consensus.cc:359] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.590906 1178 raft_consensus.cc:385] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.590984 1178 raft_consensus.cc:740] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.591065 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 710e99b4116744289f9a2130f65f4b94 with delete_type TABLET_DATA_DELETED (replaced by tablet b4ceeaba655542dab2556d3988b6e841) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.591286 1178 consensus_queue.cc:260] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.591920 1178 ts_tablet_manager.cc:1434] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:02.592311 1178 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.593653 1177 ts_tablet_manager.cc:1434] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:02.594532 1178 tablet_bootstrap.cc:654] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.594762 1177 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.595186 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 710e99b4116744289f9a2130f65f4b94 with delete_type TABLET_DATA_DELETED (replaced by tablet b4ceeaba655542dab2556d3988b6e841) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.595608 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 710e99b4116744289f9a2130f65f4b94 with delete_type TABLET_DATA_DELETED (replaced by tablet b4ceeaba655542dab2556d3988b6e841) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.595865 1176 tablet_replica.cc:333] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.596086 1176 raft_consensus.cc:2243] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.596254 1176 raft_consensus.cc:2272] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.597039 1176 ts_tablet_manager.cc:1916] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.597051 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 04b17cb6f63a4064873d0851f4b563cb
I20251029 08:52:02.597523 591 catalog_manager.cc:6719] ReplaceTablet: tablet 04b17cb6f63a4064873d0851f4b563cb deleted and replaced by tablet 61ff3274c7104c73bd35768c229833b1
I20251029 08:52:02.597725 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 710e99b4116744289f9a2130f65f4b94 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.598282 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f067b822946a45ea946eeabd484c2975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.598793 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f067b822946a45ea946eeabd484c2975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.600709 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_75195faaf53b406c889072cd3a86bba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.601208 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75195faaf53b406c889072cd3a86bba0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.602623 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b17cb6f63a4064873d0851f4b563cb with delete_type TABLET_DATA_DELETED (replaced by tablet 61ff3274c7104c73bd35768c229833b1) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.602874 1175 tablet_replica.cc:333] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.603030 1175 raft_consensus.cc:2243] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.603217 1175 raft_consensus.cc:2272] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:02.603518 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 710e99b4116744289f9a2130f65f4b94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 710e99b4116744289f9a2130f65f4b94 already in progress: creating tablet
I20251029 08:52:02.603976 1175 ts_tablet_manager.cc:1916] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.604504 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b17cb6f63a4064873d0851f4b563cb with delete_type TABLET_DATA_DELETED (replaced by tablet 61ff3274c7104c73bd35768c229833b1) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.607409 1192 raft_consensus.cc:493] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.607590 1192 raft_consensus.cc:515] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.608186 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b17cb6f63a4064873d0851f4b563cb with delete_type TABLET_DATA_DELETED (replaced by tablet 61ff3274c7104c73bd35768c229833b1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.609686 1192 leader_election.cc:290] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:02.614228 590 catalog_manager.cc:5649] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:02.615128 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]). Expected version 2 got 0
I20251029 08:52:02.624657 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e136716a2344ee8a7ada59e397bb92" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:02.625095 784 raft_consensus.cc:2468] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:02.626026 1175 ts_tablet_manager.cc:1929] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.626157 1175 log.cc:1199] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/04b17cb6f63a4064873d0851f4b563cb
I20251029 08:52:02.626561 1175 ts_tablet_manager.cc:1950] T 04b17cb6f63a4064873d0851f4b563cb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.626506 998 leader_election.cc:304] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:02.627619 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_0377c0d70fc04e3b997daf88097cb74b [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:02.628736 1192 raft_consensus.cc:2804] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:02.628870 1192 raft_consensus.cc:493] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:02.628963 1192 raft_consensus.cc:3060] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.629283 1176 ts_tablet_manager.cc:1929] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.629436 1176 log.cc:1199] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/710e99b4116744289f9a2130f65f4b94
I20251029 08:52:02.629773 1176 ts_tablet_manager.cc:1950] T 710e99b4116744289f9a2130f65f4b94 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.630687 1176 tablet_replica.cc:333] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.630890 1176 raft_consensus.cc:2243] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:02.631119 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 710e99b4116744289f9a2130f65f4b94 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:02.631342 1176 pending_rounds.cc:70] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:02.631572 1176 pending_rounds.cc:77] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037570615619584 op_type: NO_OP noop_request { }
I20251029 08:52:02.631773 1176 raft_consensus.cc:2889] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:02.631899 1176 raft_consensus.cc:2272] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.633666 1177 tablet_bootstrap.cc:654] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.635902 1176 ts_tablet_manager.cc:1916] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.644248 1177 tablet_bootstrap.cc:492] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.644438 1177 ts_tablet_manager.cc:1403] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20251029 08:52:02.644383 1192 raft_consensus.cc:515] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.645221 1177 raft_consensus.cc:359] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.645432 1177 raft_consensus.cc:385] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.645522 1177 raft_consensus.cc:740] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.645704 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e136716a2344ee8a7ada59e397bb92" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:02.645792 1177 consensus_queue.cc:260] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.645877 784 raft_consensus.cc:3060] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.646360 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b4a3272a7a0a419dadd2cad132e996fd
I20251029 08:52:02.648527 903 tablet_service.cc:1552] Processing DeleteTablet for tablet b4a3272a7a0a419dadd2cad132e996fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.648864 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aa73cfd89a724aada8af426a56b3481a
I20251029 08:52:02.649070 784 raft_consensus.cc:2468] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:02.649156 1177 ts_tablet_manager.cc:1434] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:02.649776 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b204b7d00e484f5ab89321e6f3fcb58f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.650533 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b204b7d00e484f5ab89321e6f3fcb58f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.650717 1177 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.650707 998 leader_election.cc:304] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:02.652161 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e136716a2344ee8a7ada59e397bb92" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:02.652999 903 tablet_service.cc:1552] Processing DeleteTablet for tablet aa73cfd89a724aada8af426a56b3481a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.652770 996 leader_election.cc:343] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:02.654471 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f65f8ca443e48a48d7ef4a474dc95b9 has no consensus state
I20251029 08:52:02.654508 761 tablet_service.cc:1552] Processing DeleteTablet for tablet b4a3272a7a0a419dadd2cad132e996fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.654752 763 tablet_service.cc:1552] Processing DeleteTablet for tablet aa73cfd89a724aada8af426a56b3481a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.654896 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 728e64abaadc46598bf2dbc32dc55bdd
I20251029 08:52:02.655228 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_221ab665c3fc433288528a7dcc29c8f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.655647 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9df4bbf8684046b9876226a4a342654c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.655783 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.655820 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_517bd3e187ef44cb9a149605bbdcaebd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.656196 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9df4bbf8684046b9876226a4a342654c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:02.656347 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_517bd3e187ef44cb9a149605bbdcaebd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:02.656585 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_221ab665c3fc433288528a7dcc29c8f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.656831 1183 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.656986 1183 ts_tablet_manager.cc:1403] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.054s
I20251029 08:52:02.657790 1183 raft_consensus.cc:359] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.658025 1183 raft_consensus.cc:385] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.658104 1183 raft_consensus.cc:740] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.658254 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.658378 1183 consensus_queue.cc:260] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.659144 1177 tablet_bootstrap.cc:654] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.660589 1176 ts_tablet_manager.cc:1929] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:02.660728 1176 log.cc:1199] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/04b17cb6f63a4064873d0851f4b563cb
I20251029 08:52:02.661108 1176 ts_tablet_manager.cc:1950] T 04b17cb6f63a4064873d0851f4b563cb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.662554 1176 tablet_replica.cc:333] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.662744 1176 raft_consensus.cc:2243] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.662930 1176 raft_consensus.cc:2272] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.663682 1176 ts_tablet_manager.cc:1916] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.663806 1177 tablet_bootstrap.cc:492] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.664232 1177 ts_tablet_manager.cc:1403] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20251029 08:52:02.664410 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_0377c0d70fc04e3b997daf88097cb74b [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
W20251029 08:52:02.664839 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b4a3272a7a0a419dadd2cad132e996fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4a3272a7a0a419dadd2cad132e996fd already in progress: creating tablet
I20251029 08:52:02.665071 1177 raft_consensus.cc:359] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.665264 1178 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
W20251029 08:52:02.665252 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aa73cfd89a724aada8af426a56b3481a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa73cfd89a724aada8af426a56b3481a already in progress: creating tablet
I20251029 08:52:02.665269 1177 raft_consensus.cc:385] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.665338 1177 raft_consensus.cc:740] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.665421 1178 ts_tablet_manager.cc:1403] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.034s
I20251029 08:52:02.665668 1177 consensus_queue.cc:260] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.666257 1177 ts_tablet_manager.cc:1434] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:02.666247 1178 raft_consensus.cc:359] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.666420 1178 raft_consensus.cc:385] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.666507 1178 raft_consensus.cc:740] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.666615 1177 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.666743 1178 consensus_queue.cc:260] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.667620 1178 ts_tablet_manager.cc:1434] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.667917 1178 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.669379 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet b4a3272a7a0a419dadd2cad132e996fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.669768 1178 tablet_bootstrap.cc:654] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.669484 1177 tablet_bootstrap.cc:654] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:02.670642 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b4a3272a7a0a419dadd2cad132e996fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4a3272a7a0a419dadd2cad132e996fd already in progress: creating tablet
W20251029 08:52:02.671422 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa73cfd89a724aada8af426a56b3481a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa73cfd89a724aada8af426a56b3481a already in progress: creating tablet
W20251029 08:52:02.671938 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 728e64abaadc46598bf2dbc32dc55bdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728e64abaadc46598bf2dbc32dc55bdd already in progress: creating tablet
I20251029 08:52:02.672469 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet aa73cfd89a724aada8af426a56b3481a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.673092 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.673908 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.677150 1176 ts_tablet_manager.cc:1929] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.677268 1176 log.cc:1199] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/728e64abaadc46598bf2dbc32dc55bdd
I20251029 08:52:02.677635 1176 ts_tablet_manager.cc:1950] T 728e64abaadc46598bf2dbc32dc55bdd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.678439 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:02.678612 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 728e64abaadc46598bf2dbc32dc55bdd (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
I20251029 08:52:02.679015 593 catalog_manager.cc:6719] ReplaceTablet: tablet f7f8ae2eeabd419b8999372e6aa922ae deleted and replaced by tablet 06c08679f1e746538bfa2c80d4fbdba3
I20251029 08:52:02.679283 763 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae with delete_type TABLET_DATA_DELETED (replaced by tablet 06c08679f1e746538bfa2c80d4fbdba3) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.679380 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_076b5e5e98044d93b9c0d75227b7e0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.679590 1176 tablet_replica.cc:333] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.679523 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7a0da910696e45fd9559acf90e71cde8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.679762 1176 raft_consensus.cc:2243] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.679909 1176 raft_consensus.cc:2272] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:02.680109 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.680668 1176 ts_tablet_manager.cc:1916] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.681519 903 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae with delete_type TABLET_DATA_DELETED (replaced by tablet 06c08679f1e746538bfa2c80d4fbdba3) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.681588 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a0da910696e45fd9559acf90e71cde8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:02.682169 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f7f8ae2eeabd419b8999372e6aa922ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7f8ae2eeabd419b8999372e6aa922ae already in progress: creating tablet
I20251029 08:52:02.683012 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.683601 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.688489 1176 ts_tablet_manager.cc:1929] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.688642 1176 log.cc:1199] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:02.688844 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0ee7a5a203a24a049f5a8ef903dd6aff" } new_table_name: "default.Table_f0f1005ce091415fbdb78d06887bedb5" modify_external_catalogs: true
I20251029 08:52:02.689005 1176 ts_tablet_manager.cc:1950] T f7f8ae2eeabd419b8999372e6aa922ae P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.690040 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet f7f8ae2eeabd419b8999372e6aa922ae (table default.Table_ffe5e7903bd1497391ecdeca67dae602 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:02.692134 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae with delete_type TABLET_DATA_DELETED (replaced by tablet 06c08679f1e746538bfa2c80d4fbdba3) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.698031 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]). Expected version 2 got 0
I20251029 08:52:02.699772 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 24826ee22d284341ac8ad0258ccfe567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:01 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.700102 1175 tablet_replica.cc:333] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.700330 1175 raft_consensus.cc:2243] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.700558 1175 raft_consensus.cc:2272] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.701493 1175 ts_tablet_manager.cc:1916] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.704878 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.705122 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.707202 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 7987fb4b242c4f2e80b0ec861999f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.707485 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b17cb6f63a4064873d0851f4b563cb with delete_type TABLET_DATA_DELETED (replaced by tablet 61ff3274c7104c73bd35768c229833b1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.707731 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.710952 1175 ts_tablet_manager.cc:1929] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.713485 1192 leader_election.cc:290] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:02.713786 1192 raft_consensus.cc:2804] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:02.713928 1192 raft_consensus.cc:697] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
W20251029 08:52:02.719192 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1557722a7c8041c88ee5321c0944e234 has no consensus state
I20251029 08:52:02.714358 1192 consensus_queue.cc:237] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.584954 1179 ts_tablet_manager.cc:1929] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.729938 1179 log.cc:1199] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9ec8cb95194e4a7fb0002b6db8b9073a
I20251029 08:52:02.715039 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e136716a2344ee8a7ada59e397bb92" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:02.731372 1179 ts_tablet_manager.cc:1950] T 9ec8cb95194e4a7fb0002b6db8b9073a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:02.731600 996 leader_election.cc:343] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.731774 1175 log.cc:1199] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:02.732322 1175 ts_tablet_manager.cc:1950] T 24826ee22d284341ac8ad0258ccfe567 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.733994 1175 tablet_replica.cc:333] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.734098 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 24826ee22d284341ac8ad0258ccfe567 (table default.Table_ec5e17a240a046d78a1c65dde9e90397 [id=2e26f7325f4640479006670b114d0f59]) successfully deleted
I20251029 08:52:02.734334 1176 tablet_replica.cc:333] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.735162 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9ec8cb95194e4a7fb0002b6db8b9073a (table default.Table_97135f7d4b3749e797f32a7b46656e4f [id=13774886e27e48da9bcb44cf64d42e19]) successfully deleted
I20251029 08:52:02.735600 1179 tablet_replica.cc:333] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:02.736507 1176 raft_consensus.cc:2243] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.736814 1176 raft_consensus.cc:2272] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.737730 1176 ts_tablet_manager.cc:1916] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.737998 1175 raft_consensus.cc:2243] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.738250 1175 raft_consensus.cc:2272] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.744356 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 49cb49df9bee421188997cde00967852 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f65f8ca443e48a48d7ef4a474dc95b9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.744627 759 tablet_service.cc:1552] Processing DeleteTablet for tablet b4a3272a7a0a419dadd2cad132e996fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.745198 1179 raft_consensus.cc:2243] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:02.746351 759 tablet_service.cc:1505] Processing CreateTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 (DEFAULT_TABLE table=default.Table_d881df3a5f1043079e80ac59d710e3ac [id=fe2e71ba07f34b5c9b68619573db2440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.746774 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.746613 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 95913.4us
I20251029 08:52:02.751076 1179 pending_rounds.cc:70] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:02.751374 1179 pending_rounds.cc:77] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037571503648768 op_type: NO_OP noop_request { }
I20251029 08:52:02.751565 1179 raft_consensus.cc:2889] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:02.751749 1179 raft_consensus.cc:2272] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.752250 1183 ts_tablet_manager.cc:1434] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.095s user 0.018s sys 0.026s
I20251029 08:52:02.753577 1183 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.755883 1183 tablet_bootstrap.cc:654] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.756624 1179 ts_tablet_manager.cc:1916] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.759179 759 tablet_service.cc:1505] Processing CreateTablet for tablet 1557722a7c8041c88ee5321c0944e234 (DEFAULT_TABLE table=default.Table_f0f1005ce091415fbdb78d06887bedb5 [id=224c13af437a46fab3f2bd9b3d20f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.759313 902 tablet_service.cc:1505] Processing CreateTablet for tablet 1557722a7c8041c88ee5321c0944e234 (DEFAULT_TABLE table=default.Table_f0f1005ce091415fbdb78d06887bedb5 [id=224c13af437a46fab3f2bd9b3d20f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.759627 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1557722a7c8041c88ee5321c0944e234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.759794 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1557722a7c8041c88ee5321c0944e234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.760195 1175 ts_tablet_manager.cc:1916] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.760246 761 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.760571 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9a18c9e63d4b5e9a42fe053b187b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.746246 903 tablet_service.cc:1505] Processing CreateTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 (DEFAULT_TABLE table=default.Table_d881df3a5f1043079e80ac59d710e3ac [id=fe2e71ba07f34b5c9b68619573db2440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.764346 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.767194 1178 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.767349 1178 ts_tablet_manager.cc:1403] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.099s user 0.029s sys 0.034s
I20251029 08:52:02.767431 1175 ts_tablet_manager.cc:1929] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.767554 1175 log.cc:1199] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/49cb49df9bee421188997cde00967852
I20251029 08:52:02.767959 1175 ts_tablet_manager.cc:1950] T 49cb49df9bee421188997cde00967852 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.768357 1178 raft_consensus.cc:359] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.768582 1178 raft_consensus.cc:385] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.768667 1178 raft_consensus.cc:740] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.769038 1178 consensus_queue.cc:260] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.769389 905 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.769546 1178 ts_tablet_manager.cc:1434] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:02.769459 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b204b7d00e484f5ab89321e6f3fcb58f" } new_table_name: "default.Table_fd8dcfa769f84911b4151b2c2fa503e7" modify_external_catalogs: true
I20251029 08:52:02.769775 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9a18c9e63d4b5e9a42fe053b187b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.770112 1178 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.771970 1178 tablet_bootstrap.cc:654] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.774075 1176 ts_tablet_manager.cc:1929] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.774199 1176 log.cc:1199] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7987fb4b242c4f2e80b0ec861999f405
I20251029 08:52:02.774596 1176 ts_tablet_manager.cc:1950] T 7987fb4b242c4f2e80b0ec861999f405 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.775648 1176 tablet_replica.cc:333] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.775851 1176 raft_consensus.cc:2243] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.775802 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7987fb4b242c4f2e80b0ec861999f405 (table default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]) successfully deleted
W20251029 08:52:02.776127 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 04b17cb6f63a4064873d0851f4b563cb because the tablet was not found. No further retry: Not found: Tablet not found: 04b17cb6f63a4064873d0851f4b563cb
W20251029 08:52:02.778704 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aea79bd4e6844b7abac623824cabbeeb has no consensus state
I20251029 08:52:02.779856 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9487a5ebf7ac404695ef85848278ec8d (DEFAULT_TABLE table=default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.780347 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9487a5ebf7ac404695ef85848278ec8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.781867 899 tablet_service.cc:1505] Processing CreateTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (DEFAULT_TABLE table=default.Table_2520e1f6484548089ec0f601b0bd7cf9 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.782274 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f65f8ca443e48a48d7ef4a474dc95b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.782781 904 tablet_service.cc:1505] Processing CreateTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf (DEFAULT_TABLE table=default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.783206 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a709caf6b7244a2d8ab45f6f578cf7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.785053 1175 tablet_replica.cc:333] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.785262 1175 raft_consensus.cc:2243] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.785454 1175 raft_consensus.cc:2272] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.776010 1176 raft_consensus.cc:2272] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.787770 1176 ts_tablet_manager.cc:1916] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.788832 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0377c0d70fc04e3b997daf88097cb74b" } new_table_name: "default.Table_3a2964132be042848bb5e8b54e2274d1" modify_external_catalogs: true
I20251029 08:52:02.789577 1175 ts_tablet_manager.cc:1916] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.778031 759 tablet_service.cc:1552] Processing DeleteTablet for tablet aa73cfd89a724aada8af426a56b3481a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.790664 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 49cb49df9bee421188997cde00967852 (table default.Table_2520e1f6484548089ec0f601b0bd7cf9 [id=9b51cfa4d70542a280cc72c9528c45d4]) successfully deleted
I20251029 08:52:02.793097 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c" } modify_external_catalogs: true
I20251029 08:52:02.793296 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c" } modify_external_catalogs: true
I20251029 08:52:02.794016 1176 ts_tablet_manager.cc:1929] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.794162 1176 log.cc:1199] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/62e136716a2344ee8a7ada59e397bb92
I20251029 08:52:02.794525 1176 ts_tablet_manager.cc:1950] T 62e136716a2344ee8a7ada59e397bb92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.795558 1176 tablet_replica.cc:333] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.795755 1176 raft_consensus.cc:2243] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.795899 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 62e136716a2344ee8a7ada59e397bb92 (table default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]) successfully deleted
I20251029 08:52:02.795945 1176 raft_consensus.cc:2272] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:02.796599 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61ff3274c7104c73bd35768c229833b1 has no consensus state
I20251029 08:52:02.798476 1178 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.798621 1178 ts_tablet_manager.cc:1403] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.002s
I20251029 08:52:02.799402 1178 raft_consensus.cc:359] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.799907 1178 raft_consensus.cc:385] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.800009 1178 raft_consensus.cc:740] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.800086 1176 ts_tablet_manager.cc:1916] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.800333 1178 consensus_queue.cc:260] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.801672 1178 ts_tablet_manager.cc:1434] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:02.802019 1178 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.757818 1177 tablet_bootstrap.cc:492] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.803484 1177 ts_tablet_manager.cc:1403] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.028s
I20251029 08:52:02.804468 1177 raft_consensus.cc:359] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.805289 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_087d939fe42c43c8baef8a273f08eda5" } new_table_name: "default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3" modify_external_catalogs: true
I20251029 08:52:02.806082 1175 ts_tablet_manager.cc:1929] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.806219 1175 log.cc:1199] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7987fb4b242c4f2e80b0ec861999f405
I20251029 08:52:02.806779 1175 ts_tablet_manager.cc:1950] T 7987fb4b242c4f2e80b0ec861999f405 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.806872 1176 ts_tablet_manager.cc:1929] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.807003 1176 log.cc:1199] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/49cb49df9bee421188997cde00967852
I20251029 08:52:02.807368 1176 ts_tablet_manager.cc:1950] T 49cb49df9bee421188997cde00967852 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.807905 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7987fb4b242c4f2e80b0ec861999f405 (table default.Table_98a6626e61584e298b46fadecb445bf0 [id=1a5cbd5c3f1b4e33b9feba4682923b5b]) successfully deleted
I20251029 08:52:02.808291 1176 tablet_replica.cc:333] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.808511 1176 raft_consensus.cc:2243] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.808693 1176 raft_consensus.cc:2272] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.808970 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 49cb49df9bee421188997cde00967852 (table default.Table_2520e1f6484548089ec0f601b0bd7cf9 [id=9b51cfa4d70542a280cc72c9528c45d4]) successfully deleted
I20251029 08:52:02.809147 1178 tablet_bootstrap.cc:654] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.809453 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a2cb1179d9de4d818e95e90c74afef35" } modify_external_catalogs: true
I20251029 08:52:02.809492 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b8068c1c7ce49fc85b660f40a7d1c58 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.809706 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a2cb1179d9de4d818e95e90c74afef35" } modify_external_catalogs: true
I20251029 08:52:02.810357 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]). Expected version 1 got 0
I20251029 08:52:02.811275 590 master_service.cc:954] ReplaceTablet: received request to replace tablet f7f8ae2eeabd419b8999372e6aa922ae from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.812935 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]). Expected version 1 got 0
I20251029 08:52:02.813369 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9487a5ebf7ac404695ef85848278ec8d
I20251029 08:52:02.780843 763 tablet_service.cc:1505] Processing CreateTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (DEFAULT_TABLE table=default.Table_2520e1f6484548089ec0f601b0bd7cf9 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.814453 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f65f8ca443e48a48d7ef4a474dc95b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.814466 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 9487a5ebf7ac404695ef85848278ec8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.815109 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9487a5ebf7ac404695ef85848278ec8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9487a5ebf7ac404695ef85848278ec8d already in progress: creating tablet
I20251029 08:52:02.816324 1178 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.816644 1178 ts_tablet_manager.cc:1403] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251029 08:52:02.817485 1178 raft_consensus.cc:359] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:02.817618 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 9487a5ebf7ac404695ef85848278ec8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.817708 1178 raft_consensus.cc:385] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.817786 1178 raft_consensus.cc:740] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.779481 761 tablet_service.cc:1505] Processing CreateTablet for tablet 9487a5ebf7ac404695ef85848278ec8d (DEFAULT_TABLE table=default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:02.818113 582 catalog_manager.cc:5101] could not convert tablet's e9af06b4d79441968bda6bd7d67d76bf schema from PB: OK
I20251029 08:52:02.818080 1178 consensus_queue.cc:260] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:02.818418 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9487a5ebf7ac404695ef85848278ec8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.818782 1178 ts_tablet_manager.cc:1434] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:02.819171 1178 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.822111 1178 tablet_bootstrap.cc:654] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.804658 1177 raft_consensus.cc:385] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.824240 1177 raft_consensus.cc:740] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.824869 1176 ts_tablet_manager.cc:1916] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.826296 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4b8068c1c7ce49fc85b660f40a7d1c58
I20251029 08:52:02.826768 593 catalog_manager.cc:6719] ReplaceTablet: tablet 4b8068c1c7ce49fc85b660f40a7d1c58 deleted and replaced by tablet 453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:02.826949 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:02.827564 593 master_service.cc:954] ReplaceTablet: received request to replace tablet e9af06b4d79441968bda6bd7d67d76bf from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.828131 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b8068c1c7ce49fc85b660f40a7d1c58 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.829200 758 tablet_service.cc:1505] Processing CreateTablet for tablet a232a4a898f04f339084c807990a14e1 (DEFAULT_TABLE table=default.Table_35f72924484241fbaf61c2cafa158f4e [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.829348 762 tablet_service.cc:1505] Processing CreateTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf (DEFAULT_TABLE table=default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.829612 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a232a4a898f04f339084c807990a14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.829758 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet 453911c5a29f43e7a21fa2c47ef264e8) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.829934 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.830122 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a709caf6b7244a2d8ab45f6f578cf7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.832172 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.832759 1190 raft_consensus.cc:493] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.832924 1190 raft_consensus.cc:515] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.833593 1190 leader_election.cc:290] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.833513 904 tablet_service.cc:1505] Processing CreateTablet for tablet 8ded7b3943e544e585fe52433ff7555f (DEFAULT_TABLE table=default.Table_087d939fe42c43c8baef8a273f08eda5 [id=5124fdf717764856bff708bedd9656c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.833942 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ded7b3943e544e585fe52433ff7555f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.834070 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ded7b3943e544e585fe52433ff7555f consensus state has no leader
W20251029 08:52:02.834458 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 710e99b4116744289f9a2130f65f4b94 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 710e99b4116744289f9a2130f65f4b94 consensus state has no leader
I20251029 08:52:02.835875 903 tablet_service.cc:1505] Processing CreateTablet for tablet a232a4a898f04f339084c807990a14e1 (DEFAULT_TABLE table=default.Table_35f72924484241fbaf61c2cafa158f4e [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.836326 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a232a4a898f04f339084c807990a14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.838949 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa73cfd89a724aada8af426a56b3481a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:02.839738 1177 consensus_queue.cc:260] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:02.840471 718 leader_election.cc:343] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:02.840502 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0d8a59f8f764466686a4b25c3b28596f because the tablet was not found. No further retry: Not found: Tablet not found: 0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:02.844177 900 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a59f8f764466686a4b25c3b28596f (DEFAULT_TABLE table=default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.844774 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a59f8f764466686a4b25c3b28596f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.844910 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:02.845716 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 62e136716a2344ee8a7ada59e397bb92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e136716a2344ee8a7ada59e397bb92 already in progress: creating tablet
I20251029 08:52:02.847046 1177 ts_tablet_manager.cc:1434] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.043s user 0.000s sys 0.002s
I20251029 08:52:02.847714 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e9af06b4d79441968bda6bd7d67d76bf
I20251029 08:52:02.848222 593 catalog_manager.cc:6719] ReplaceTablet: tablet e9af06b4d79441968bda6bd7d67d76bf deleted and replaced by tablet 13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:02.849117 898 tablet_service.cc:1552] Processing DeleteTablet for tablet e9af06b4d79441968bda6bd7d67d76bf with delete_type TABLET_DATA_DELETED (replaced by tablet 13eeb2e8951b421e9d6f8f228a956e3d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.849440 1175 tablet_replica.cc:333] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.849668 1175 raft_consensus.cc:2243] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.849897 1175 raft_consensus.cc:2272] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.851058 1175 ts_tablet_manager.cc:1916] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.854760 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 62e136716a2344ee8a7ada59e397bb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.855243 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd" } modify_external_catalogs: true
I20251029 08:52:02.855458 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd" } modify_external_catalogs: true
I20251029 08:52:02.858043 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2520e1f6484548089ec0f601b0bd7cf9" } new_table_name: "default.Table_7966c1de42494873aa075b0d9185824d" modify_external_catalogs: true
I20251029 08:52:02.858690 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3" } modify_external_catalogs: true
I20251029 08:52:02.858907 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3" } modify_external_catalogs: true
I20251029 08:52:02.859398 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f067b822946a45ea946eeabd484c2975" } modify_external_catalogs: true
I20251029 08:52:02.859582 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f067b822946a45ea946eeabd484c2975" } modify_external_catalogs: true
I20251029 08:52:02.867812 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a709caf6b7244a2d8ab45f6f578cf7cf
I20251029 08:52:02.871371 758 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a59f8f764466686a4b25c3b28596f (DEFAULT_TABLE table=default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.871826 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a59f8f764466686a4b25c3b28596f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.872080 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f65f8ca443e48a48d7ef4a474dc95b9 consensus state has no leader
I20251029 08:52:02.873057 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.873878 763 tablet_service.cc:1505] Processing CreateTablet for tablet 8ded7b3943e544e585fe52433ff7555f (DEFAULT_TABLE table=default.Table_087d939fe42c43c8baef8a273f08eda5 [id=5124fdf717764856bff708bedd9656c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.874675 760 tablet_service.cc:1552] Processing DeleteTablet for tablet e9af06b4d79441968bda6bd7d67d76bf with delete_type TABLET_DATA_DELETED (replaced by tablet 13eeb2e8951b421e9d6f8f228a956e3d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.874913 1190 raft_consensus.cc:493] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.875059 1190 raft_consensus.cc:515] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.875957 1190 leader_election.cc:290] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.876431 760 tablet_service.cc:1552] Processing DeleteTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:02.876742 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:02.880661 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet 453911c5a29f43e7a21fa2c47ef264e8) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.882675 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4ad4591d3d494b20a6f4f88479c97229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.882980 718 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:02.883288 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ad4591d3d494b20a6f4f88479c97229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.884101 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ded7b3943e544e585fe52433ff7555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.885725 1177 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.888090 1175 ts_tablet_manager.cc:1929] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.888231 1175 log.cc:1199] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e9af06b4d79441968bda6bd7d67d76bf
I20251029 08:52:02.888667 1175 ts_tablet_manager.cc:1950] T e9af06b4d79441968bda6bd7d67d76bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.890105 1177 tablet_bootstrap.cc:654] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.892818 590 master_service.cc:954] ReplaceTablet: received request to replace tablet a709caf6b7244a2d8ab45f6f578cf7cf from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:02.892931 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 (DEFAULT_TABLE table=default.Table_d881df3a5f1043079e80ac59d710e3ac [id=fe2e71ba07f34b5c9b68619573db2440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.893529 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.894034 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_70788044eecf4c7da477e61f7610e1c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.894034 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a7141f1ff12e4dbfae563a27088ad272"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.894603 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70788044eecf4c7da477e61f7610e1c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:02.894745 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7141f1ff12e4dbfae563a27088ad272 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.895886 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:02.896034 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (DEFAULT_TABLE table=default.Table_2520e1f6484548089ec0f601b0bd7cf9 [id=9b51cfa4d70542a280cc72c9528c45d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.896279 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa73cfd89a724aada8af426a56b3481a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:02.896481 1071 raft_consensus.cc:2468] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:02.896577 1238 raft_consensus.cc:493] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:02.897178 1175 tablet_replica.cc:333] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.897182 1183 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:02.897343 1183 ts_tablet_manager.cc:1403] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.000s
I20251029 08:52:02.897264 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f1c66f1bd2cd4518ac472eca739914f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.897380 1175 raft_consensus.cc:2243] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.897557 1175 raft_consensus.cc:2272] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:02.897748 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c66f1bd2cd4518ac472eca739914f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.898900 1183 raft_consensus.cc:359] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.899106 1183 raft_consensus.cc:385] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.899626 1183 raft_consensus.cc:740] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.899585 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_97621cac05024570b4f5355e6ea12e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.899890 1183 consensus_queue.cc:260] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:02.900314 718 consensus_peers.cc:597] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9af06b4d79441968bda6bd7d67d76bf. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:02.900259 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_69a6a8b8bfed4039a0833b799b013493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.900449 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 9487a5ebf7ac404695ef85848278ec8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:02.900727 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a6a8b8bfed4039a0833b799b013493 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.901222 1175 ts_tablet_manager.cc:1916] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.901407 720 leader_election.cc:304] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.901949 1190 raft_consensus.cc:2804] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:02.902077 1190 raft_consensus.cc:493] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:02.902190 1190 raft_consensus.cc:3060] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.902566 1176 ts_tablet_manager.cc:1929] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.902717 1176 log.cc:1199] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b4a3272a7a0a419dadd2cad132e996fd
W20251029 08:52:02.902920 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97621cac05024570b4f5355e6ea12e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.903111 1176 ts_tablet_manager.cc:1950] T b4a3272a7a0a419dadd2cad132e996fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.904127 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b4a3272a7a0a419dadd2cad132e996fd (table default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]) successfully deleted
I20251029 08:52:02.904388 1183 ts_tablet_manager.cc:1434] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:02.904865 1178 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.905009 1178 ts_tablet_manager.cc:1403] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.014s
I20251029 08:52:02.905825 1178 raft_consensus.cc:359] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.906015 1178 raft_consensus.cc:385] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.906025 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.906100 1178 raft_consensus.cc:740] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.906344 1178 consensus_queue.cc:260] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.906936 1178 ts_tablet_manager.cc:1434] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.907598 1178 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.908550 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 8ded7b3943e544e585fe52433ff7555f (DEFAULT_TABLE table=default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3 [id=5124fdf717764856bff708bedd9656c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.909614 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a59f8f764466686a4b25c3b28596f (DEFAULT_TABLE table=default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.910677 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet e9af06b4d79441968bda6bd7d67d76bf with delete_type TABLET_DATA_DELETED (replaced by tablet 13eeb2e8951b421e9d6f8f228a956e3d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.911134 1177 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.911206 1176 tablet_replica.cc:333] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.911298 1177 ts_tablet_manager.cc:1403] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20251029 08:52:02.911444 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.911567 1178 tablet_bootstrap.cc:654] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.912240 1183 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:02.912598 1177 raft_consensus.cc:359] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.912849 1177 raft_consensus.cc:385] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.912863 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 9487a5ebf7ac404695ef85848278ec8d (DEFAULT_TABLE table=default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.912940 1177 raft_consensus.cc:740] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.913213 1047 tablet_service.cc:1505] Processing CreateTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf (DEFAULT_TABLE table=default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.913314 1177 consensus_queue.cc:260] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.913542 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 1557722a7c8041c88ee5321c0944e234 (DEFAULT_TABLE table=default.Table_f0f1005ce091415fbdb78d06887bedb5 [id=224c13af437a46fab3f2bd9b3d20f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.913730 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.914260 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet 453911c5a29f43e7a21fa2c47ef264e8) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:02.914640 1048 tablet_service.cc:1505] Processing CreateTablet for tablet a232a4a898f04f339084c807990a14e1 (DEFAULT_TABLE table=default.Table_35f72924484241fbaf61c2cafa158f4e [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.915885 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ded7b3943e544e585fe52433ff7555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.916081 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a709caf6b7244a2d8ab45f6f578cf7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.917022 1177 ts_tablet_manager.cc:1434] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:02.918478 1177 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.919566 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9487a5ebf7ac404695ef85848278ec8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.920132 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a232a4a898f04f339084c807990a14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.920711 1178 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.920866 1178 ts_tablet_manager.cc:1403] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20251029 08:52:02.921324 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a59f8f764466686a4b25c3b28596f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.921947 1178 raft_consensus.cc:359] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.922139 1178 raft_consensus.cc:385] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.922358 1178 raft_consensus.cc:740] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.922614 1178 consensus_queue.cc:260] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.923851 1178 ts_tablet_manager.cc:1434] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:02.924168 1178 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.924190 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1557722a7c8041c88ee5321c0944e234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.924950 1175 ts_tablet_manager.cc:1929] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.925079 1175 log.cc:1199] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/62e136716a2344ee8a7ada59e397bb92
I20251029 08:52:02.925412 1175 ts_tablet_manager.cc:1950] T 62e136716a2344ee8a7ada59e397bb92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:02.925840 1178 tablet_bootstrap.cc:654] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.926479 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 62e136716a2344ee8a7ada59e397bb92 (table default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]) successfully deleted
I20251029 08:52:02.927942 1177 tablet_bootstrap.cc:654] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.928170 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9a18c9e63d4b5e9a42fe053b187b08. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.929819 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a709caf6b7244a2d8ab45f6f578cf7cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a709caf6b7244a2d8ab45f6f578cf7cf already in progress: creating tablet
W20251029 08:52:02.930503 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b8068c1c7ce49fc85b660f40a7d1c58 already in progress: creating tablet
I20251029 08:52:02.931367 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f65f8ca443e48a48d7ef4a474dc95b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.933200 1178 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.933355 1178 ts_tablet_manager.cc:1403] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20251029 08:52:02.934185 1177 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:02.934310 1177 ts_tablet_manager.cc:1403] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20251029 08:52:02.934167 1178 raft_consensus.cc:359] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.934509 1178 raft_consensus.cc:385] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.934599 1178 raft_consensus.cc:740] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.934832 1178 consensus_queue.cc:260] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.935088 1177 raft_consensus.cc:359] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.935247 1177 raft_consensus.cc:385] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.935339 1177 raft_consensus.cc:740] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.935618 1178 ts_tablet_manager.cc:1434] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:02.935590 1177 consensus_queue.cc:260] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.935894 1178 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.937114 1177 ts_tablet_manager.cc:1434] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:02.937762 1178 tablet_bootstrap.cc:654] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.938419 1071 raft_consensus.cc:1275] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Refusing update from remote peer 7a8d80098fcb40ada903c3b2bacc2ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:02.940210 1177 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:02.940709 1214 consensus_queue.cc:1048] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:02.942414 1177 tablet_bootstrap.cc:654] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.948158 1190 raft_consensus.cc:515] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.948748 1176 raft_consensus.cc:2243] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.949437 1176 raft_consensus.cc:2272] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.949671 1190 leader_election.cc:290] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:02.949730 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa73cfd89a724aada8af426a56b3481a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:02.950613 718 leader_election.cc:343] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.951125 1176 ts_tablet_manager.cc:1916] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.953401 903 tablet_service.cc:1505] Processing CreateTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (DEFAULT_TABLE table=default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.953989 1178 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:02.954152 1178 ts_tablet_manager.cc:1403] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20251029 08:52:02.954511 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.955019 1178 raft_consensus.cc:359] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.955260 1178 raft_consensus.cc:385] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:02.955353 1178 raft_consensus.cc:740] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:02.955637 904 tablet_service.cc:1505] Processing CreateTablet for tablet e899e183e255498b83a6d740af98de42 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.955747 761 tablet_service.cc:1505] Processing CreateTablet for tablet e899e183e255498b83a6d740af98de42 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.956147 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e899e183e255498b83a6d740af98de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.956171 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e899e183e255498b83a6d740af98de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.957612 902 tablet_service.cc:1505] Processing CreateTablet for tablet 61ff3274c7104c73bd35768c229833b1 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.915594 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:02.957940 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ff3274c7104c73bd35768c229833b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.960340 758 tablet_service.cc:1505] Processing CreateTablet for tablet 61ff3274c7104c73bd35768c229833b1 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.960698 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ff3274c7104c73bd35768c229833b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.960968 759 tablet_service.cc:1505] Processing CreateTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (DEFAULT_TABLE table=default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.961216 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]). Expected version 1 got 0
I20251029 08:52:02.961352 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.961931 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_859570febd594b1782eaa7388665eef0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.962452 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_859570febd594b1782eaa7388665eef0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.964211 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_57a9dba12bd24aedb38524e8c99dfe84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:02.964684 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57a9dba12bd24aedb38524e8c99dfe84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.969597 905 tablet_service.cc:1505] Processing CreateTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 (DEFAULT_TABLE table=default.Table_ffe5e7903bd1497391ecdeca67dae602 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.969823 1178 consensus_queue.cc:260] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.970052 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c08679f1e746538bfa2c80d4fbdba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.970366 1178 ts_tablet_manager.cc:1434] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251029 08:52:02.971395 1178 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:02.973242 1178 tablet_bootstrap.cc:654] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:02.906133 1238 raft_consensus.cc:515] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:02.976910 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:02.977097 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:02.978186 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]). Expected version 1 got 0
I20251029 08:52:02.981070 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 1557722a7c8041c88ee5321c0944e234 (table default.Table_f0f1005ce091415fbdb78d06887bedb5 [id=224c13af437a46fab3f2bd9b3d20f523]). Expected version 1 got 0
I20251029 08:52:02.983713 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_823727b7e1194e56a35d64f1d58d761a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:02.986128 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 710e99b4116744289f9a2130f65f4b94 with delete_type TABLET_DATA_DELETED (replaced by tablet b4ceeaba655542dab2556d3988b6e841) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:02.986944 761 tablet_service.cc:1505] Processing CreateTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 (DEFAULT_TABLE table=default.Table_ffe5e7903bd1497391ecdeca67dae602 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.987403 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c08679f1e746538bfa2c80d4fbdba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.988008 1175 tablet_replica.cc:333] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:02.988215 1175 raft_consensus.cc:2243] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:02.988299 1176 ts_tablet_manager.cc:1929] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:02.988395 1175 raft_consensus.cc:2272] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:02.988473 1176 log.cc:1199] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aa73cfd89a724aada8af426a56b3481a
I20251029 08:52:02.988937 1176 ts_tablet_manager.cc:1950] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:02.989164 1175 ts_tablet_manager.cc:1916] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:02.989780 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet aa73cfd89a724aada8af426a56b3481a (table default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]) successfully deleted
I20251029 08:52:02.989881 1176 tablet_replica.cc:333] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:02.990084 1176 raft_consensus.cc:2243] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:02.990195 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9487a5ebf7ac404695ef85848278ec8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9487a5ebf7ac404695ef85848278ec8d already in progress: creating tablet
I20251029 08:52:02.990269 1176 raft_consensus.cc:2272] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:02.990597 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_823727b7e1194e56a35d64f1d58d761a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:02.991263 758 tablet_service.cc:1505] Processing CreateTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d (DEFAULT_TABLE table=default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.991657 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13eeb2e8951b421e9d6f8f228a956e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:02.992092 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa73cfd89a724aada8af426a56b3481a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:02.995055 1069 raft_consensus.cc:3060] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:02.996551 763 tablet_service.cc:1505] Processing CreateTablet for tablet b4ceeaba655542dab2556d3988b6e841 (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:02.996927 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ceeaba655542dab2556d3988b6e841. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:02.998525 720 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:02.998668 720 leader_election.cc:304] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:02.999439 1190 raft_consensus.cc:2749] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.000821 905 tablet_service.cc:1505] Processing CreateTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.005601 904 tablet_service.cc:1505] Processing CreateTablet for tablet b4ceeaba655542dab2556d3988b6e841 (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.006249 903 tablet_service.cc:1505] Processing CreateTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d (DEFAULT_TABLE table=default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.007218 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bb01c123c44fb69b2f174fca5d7a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.008739 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d67ccc3d92a84bb8abec8560b542ca91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.009454 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67ccc3d92a84bb8abec8560b542ca91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.011945 899 tablet_service.cc:1505] Processing CreateTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.012331 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9cf2a8f2254a38a725bcd3c83e8509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.012285 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_221ab665c3fc433288528a7dcc29c8f8" } new_table_name: "default.Table_1c46023036d34ee7be5bb30a14caa0e2" modify_external_catalogs: true
I20251029 08:52:03.015872 900 tablet_service.cc:1505] Processing CreateTablet for tablet aea79bd4e6844b7abac623824cabbeeb (DEFAULT_TABLE table=default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.015985 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ffe5e7903bd1497391ecdeca67dae602" } new_table_name: "default.Table_2c76bf4e84c04dfca8b4897658618b06" modify_external_catalogs: true
I20251029 08:52:03.016381 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea79bd4e6844b7abac623824cabbeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.017616 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f0f1005ce091415fbdb78d06887bedb5" } new_table_name: "default.Table_3a8b8f91860e419e8538f028e57603fd" modify_external_catalogs: true
I20251029 08:52:03.020587 902 tablet_service.cc:1505] Processing CreateTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.023263 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6b8a4b96d3c748daa0f467768b2b7abd
I20251029 08:52:03.024196 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453911c5a29f43e7a21fa2c47ef264e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.024425 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8ded7b3943e544e585fe52433ff7555f
I20251029 08:52:03.027096 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13eeb2e8951b421e9d6f8f228a956e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.028301 1193 raft_consensus.cc:493] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.028491 1193 raft_consensus.cc:515] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.029138 1193 leader_election.cc:290] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.031641 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.031587 898 tablet_service.cc:1505] Processing CreateTablet for tablet 06eb710809754e7abd00a345e8ae591a (DEFAULT_TABLE table=default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.031850 897 tablet_service.cc:1505] Processing CreateTablet for tablet 25b2e13184de4c25a5f14bc78d871bda (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.032289 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06eb710809754e7abd00a345e8ae591a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.032848 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2e13184de4c25a5f14bc78d871bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.035869 901 tablet_service.cc:1505] Processing CreateTablet for tablet be476c9994ed4dc28f606955b16bc4de (DEFAULT_TABLE table=default.Table_221ab665c3fc433288528a7dcc29c8f8 [id=6c75c2c51a93469489dfd59bce7906c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.036366 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be476c9994ed4dc28f606955b16bc4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.044400 593 master_service.cc:954] ReplaceTablet: received request to replace tablet aa73cfd89a724aada8af426a56b3481a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.045042 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e0794b7ebfe242f3b2dc881492a95de6" } new_table_name: "default.Table_0567a7bfaea44a6d8bd8807e6206939c" modify_external_catalogs: true
I20251029 08:52:03.046300 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ad4591d3d494b20a6f4f88479c97229" } modify_external_catalogs: true
I20251029 08:52:03.046483 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ad4591d3d494b20a6f4f88479c97229" } modify_external_catalogs: true
I20251029 08:52:03.048424 589 master_service.cc:954] ReplaceTablet: received request to replace tablet b4a3272a7a0a419dadd2cad132e996fd from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.049858 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd (DEFAULT_TABLE table=default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.050336 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8a4b96d3c748daa0f467768b2b7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.050750 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_31cdca53404a4a36a5ee867f8aa53bfc" } modify_external_catalogs: true
I20251029 08:52:03.050935 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_31cdca53404a4a36a5ee867f8aa53bfc" } modify_external_catalogs: true
I20251029 08:52:03.051061 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.051383 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ceeaba655542dab2556d3988b6e841. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.056905 998 leader_election.cc:343] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 49cb49df9bee421188997cde00967852
I20251029 08:52:03.057595 589 master_service.cc:954] ReplaceTablet: received request to replace tablet e899e183e255498b83a6d740af98de42 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.058182 1043 tablet_service.cc:1505] Processing CreateTablet for tablet b4ceeaba655542dab2556d3988b6e841 (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.060354 1051 tablet_service.cc:1505] Processing CreateTablet for tablet e899e183e255498b83a6d740af98de42 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.064327 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d (DEFAULT_TABLE table=default.Table_e0794b7ebfe242f3b2dc881492a95de6 [id=407839d19657473883ad497873b78fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.068795 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d8a59f8f764466686a4b25c3b28596f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.070853 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_75195faaf53b406c889072cd3a86bba0" } modify_external_catalogs: true
I20251029 08:52:03.071225 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_75195faaf53b406c889072cd3a86bba0" } modify_external_catalogs: true
I20251029 08:52:03.072316 1183 tablet_bootstrap.cc:654] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.074167 1238 leader_election.cc:290] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:03.075026 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 61ff3274c7104c73bd35768c229833b1 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:03.075686 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 54432.9us
W20251029 08:52:03.075728 996 leader_election.cc:343] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 49cb49df9bee421188997cde00967852
I20251029 08:52:03.076231 996 leader_election.cc:304] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.076274 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:03.076622 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 70352.3us
W20251029 08:52:03.076844 861 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.077596 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.078336 1036 tablet_service.cc:1505] Processing CreateTablet for tablet be476c9994ed4dc28f606955b16bc4de (DEFAULT_TABLE table=default.Table_221ab665c3fc433288528a7dcc29c8f8 [id=6c75c2c51a93469489dfd59bce7906c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.078568 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd (DEFAULT_TABLE table=default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.078861 1038 tablet_service.cc:1505] Processing CreateTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.079447 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.079674 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.080065 1049 tablet_service.cc:1505] Processing CreateTablet for tablet aea79bd4e6844b7abac623824cabbeeb (DEFAULT_TABLE table=default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.080735 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 06eb710809754e7abd00a345e8ae591a (DEFAULT_TABLE table=default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.081506 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 25b2e13184de4c25a5f14bc78d871bda (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.082099 1238 raft_consensus.cc:2749] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.066639 1045 tablet_service.cc:1505] Processing CreateTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (DEFAULT_TABLE table=default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:03.084314 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be476c9994ed4dc28f606955b16bc4de consensus state has no leader
I20251029 08:52:03.084786 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_97621cac05024570b4f5355e6ea12e2d" } new_table_name: "default.Table_eedf056f49564746a9c2c87482f5db42" modify_external_catalogs: true
I20251029 08:52:03.086055 901 tablet_service.cc:1505] Processing CreateTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (DEFAULT_TABLE table=default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:03.087805 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06c08679f1e746538bfa2c80d4fbdba3 consensus state has no leader
W20251029 08:52:03.091992 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1557722a7c8041c88ee5321c0944e234 consensus state has no leader
I20251029 08:52:03.087608 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1d1d5b2a9e4051bbcd25be1578ee73. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.107566 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13eeb2e8951b421e9d6f8f228a956e3d consensus state has no leader
I20251029 08:52:03.112179 1069 raft_consensus.cc:2468] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:03.112882 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13eeb2e8951b421e9d6f8f228a956e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.113453 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ceeaba655542dab2556d3988b6e841. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.113757 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2e13184de4c25a5f14bc78d871bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.116417 897 tablet_service.cc:1505] Processing CreateTablet for tablet 14adc0f009d94287b92eda7b171c7c13 (DEFAULT_TABLE table=default.Table_f1c66f1bd2cd4518ac472eca739914f8 [id=9a65ac9ea0f448a38b2098fcefd707dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.116874 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be476c9994ed4dc28f606955b16bc4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.116853 900 tablet_service.cc:1505] Processing CreateTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 (DEFAULT_TABLE table=default.Table_69a6a8b8bfed4039a0833b799b013493 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.117771 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06eb710809754e7abd00a345e8ae591a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.120183 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453911c5a29f43e7a21fa2c47ef264e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.120769 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9cf2a8f2254a38a725bcd3c83e8509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.121574 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea79bd4e6844b7abac623824cabbeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.124326 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8a4b96d3c748daa0f467768b2b7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.128543 899 tablet_service.cc:1505] Processing CreateTablet for tablet 6e2c32b84201470aa0dd697a5da12117 (DEFAULT_TABLE table=default.Table_97621cac05024570b4f5355e6ea12e2d [id=3a1661ecd47240a79cf75ebbcaf6acf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.129149 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.130622 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 6e2c32b84201470aa0dd697a5da12117 (DEFAULT_TABLE table=default.Table_97621cac05024570b4f5355e6ea12e2d [id=3a1661ecd47240a79cf75ebbcaf6acf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.131204 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2c32b84201470aa0dd697a5da12117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.132509 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 954c264e600a46f19db40dde5180787d (DEFAULT_TABLE table=default.Table_a7141f1ff12e4dbfae563a27088ad272 [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.132786 1042 tablet_service.cc:1505] Processing CreateTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (DEFAULT_TABLE table=default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.132927 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954c264e600a46f19db40dde5180787d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.133363 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1d1d5b2a9e4051bbcd25be1578ee73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.134977 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 14adc0f009d94287b92eda7b171c7c13 (DEFAULT_TABLE table=default.Table_f1c66f1bd2cd4518ac472eca739914f8 [id=9a65ac9ea0f448a38b2098fcefd707dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.135350 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14adc0f009d94287b92eda7b171c7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.136456 898 tablet_service.cc:1505] Processing CreateTablet for tablet 954c264e600a46f19db40dde5180787d (DEFAULT_TABLE table=default.Table_a7141f1ff12e4dbfae563a27088ad272 [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.137054 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e899e183e255498b83a6d740af98de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.139875 1179 ts_tablet_manager.cc:1929] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:03.140049 1179 log.cc:1199] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/62e136716a2344ee8a7ada59e397bb92
I20251029 08:52:03.140512 1179 ts_tablet_manager.cc:1950] T 62e136716a2344ee8a7ada59e397bb92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.141717 1179 tablet_replica.cc:333] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.142020 1179 raft_consensus.cc:2243] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.142269 1179 raft_consensus.cc:2272] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.142977 1179 ts_tablet_manager.cc:1916] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.147545 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.147627 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 62e136716a2344ee8a7ada59e397bb92 (table default.Table_935e9f61644a475fa4158f5807ff839b [id=b2137c128c3b481287fe61db8b54de3f]) successfully deleted
I20251029 08:52:03.148250 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ff3274c7104c73bd35768c229833b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.152618 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bb01c123c44fb69b2f174fca5d7a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.152988 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 (DEFAULT_TABLE table=default.Table_69a6a8b8bfed4039a0833b799b013493 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.153447 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e339c9ed26a48a2a9fe172201c5cc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.154688 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e72f64830e924fa48e2fa4cf81ccbba3
I20251029 08:52:03.155606 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.154688 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet de8f868095a642f6a5d70c30bb74105c
I20251029 08:52:03.157403 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.157739 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet de8f868095a642f6a5d70c30bb74105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.159396 896 tablet_service.cc:1552] Processing DeleteTablet for tablet de8f868095a642f6a5d70c30bb74105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.160858 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cc1d1d5b2a9e4051bbcd25be1578ee73
I20251029 08:52:03.161125 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e899e183e255498b83a6d740af98de42
I20251029 08:52:03.161588 589 catalog_manager.cc:6719] ReplaceTablet: tablet e899e183e255498b83a6d740af98de42 deleted and replaced by tablet 904523f7164849139cddb1100e95b824
I20251029 08:52:03.161566 896 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.161834 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.162309 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14adc0f009d94287b92eda7b171c7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.162599 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2c32b84201470aa0dd697a5da12117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.162788 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e339c9ed26a48a2a9fe172201c5cc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.166354 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954c264e600a46f19db40dde5180787d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.170367 1177 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.172878 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b8068c1c7ce49fc85b660f40a7d1c58 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.174679 1177 ts_tablet_manager.cc:1403] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.235s user 0.003s sys 0.002s
I20251029 08:52:03.176582 1177 raft_consensus.cc:359] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.177174 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 (DEFAULT_TABLE table=default.Table_ffe5e7903bd1497391ecdeca67dae602 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.177543 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66c8696f5f854516b099c64d58891cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.178076 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66c8696f5f854516b099c64d58891cdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.179900 1176 ts_tablet_manager.cc:1916] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.180766 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:03.184190 784 raft_consensus.cc:2468] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:03.184787 720 leader_election.cc:304] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.186359 1175 ts_tablet_manager.cc:1929] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.190636 1177 raft_consensus.cc:385] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.191586 1177 raft_consensus.cc:740] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.195156 1177 consensus_queue.cc:260] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.196163 1177 ts_tablet_manager.cc:1434] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
W20251029 08:52:03.196210 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e2c32b84201470aa0dd697a5da12117 consensus state has no leader
I20251029 08:52:03.197386 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.199556 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c08679f1e746538bfa2c80d4fbdba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.199744 762 tablet_service.cc:1505] Processing CreateTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.201300 1175 log.cc:1199] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/710e99b4116744289f9a2130f65f4b94
I20251029 08:52:03.201817 1175 ts_tablet_manager.cc:1950] T 710e99b4116744289f9a2130f65f4b94 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:03.201977 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.204398 1043 tablet_service.cc:1505] Processing CreateTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 (DEFAULT_TABLE table=default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.206204 897 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f8ae2eeabd419b8999372e6aa922ae with delete_type TABLET_DATA_DELETED (replaced by tablet 06c08679f1e746538bfa2c80d4fbdba3) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.207829 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72f64830e924fa48e2fa4cf81ccbba3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.208276 720 raft_consensus.cc:2686] T aa73cfd89a724aada8af426a56b3481a P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:03.209378 861 leader_election.cc:304] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:03.210054 764 tablet_service.cc:1505] Processing CreateTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.210610 1238 raft_consensus.cc:493] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.210768 1238 raft_consensus.cc:515] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.211937 1193 raft_consensus.cc:2804] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:03.212174 1193 raft_consensus.cc:493] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:03.212339 1193 raft_consensus.cc:3060] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:03.219134 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 710e99b4116744289f9a2130f65f4b94 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:03.219378 1193 raft_consensus.cc:515] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:03.219799 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6b8a4b96d3c748daa0f467768b2b7abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b8a4b96d3c748daa0f467768b2b7abd already in progress: creating tablet
W20251029 08:52:03.220430 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8ded7b3943e544e585fe52433ff7555f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ded7b3943e544e585fe52433ff7555f already in progress: creating tablet
W20251029 08:52:03.222206 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e72f64830e924fa48e2fa4cf81ccbba3 because the tablet was not found. No further retry: Not found: Tablet not found: e72f64830e924fa48e2fa4cf81ccbba3
I20251029 08:52:03.222532 1175 tablet_replica.cc:333] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:03.222788 1175 raft_consensus.cc:2243] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.223027 1175 raft_consensus.cc:2272] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.225495 905 tablet_service.cc:1505] Processing CreateTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 (DEFAULT_TABLE table=default.Table_d67ccc3d92a84bb8abec8560b542ca91 [id=6f5f02000b50491682d2ab710c4e1f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.225963 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2dd9d6d0854e03a8298625938065c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.226606 899 tablet_service.cc:1505] Processing CreateTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 (DEFAULT_TABLE table=default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.227433 760 tablet_service.cc:1505] Processing CreateTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 (DEFAULT_TABLE table=default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.227623 897 tablet_service.cc:1505] Processing CreateTablet for tablet 6426b5a4f5784b3a97475cb23199f916 (DEFAULT_TABLE table=default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.227895 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a2b37e45e444caa3f5117ae9a25304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.228253 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6426b5a4f5784b3a97475cb23199f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.228835 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bb01c123c44fb69b2f174fca5d7a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.228966 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.229935 1248 mvcc.cc:204] Tried to move back new op lower bound from 7216037572188626944 to 7216037570017759232. Current Snapshot: MvccSnapshot[applied={T|T < 7216037572188626944}]
I20251029 08:52:03.230188 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:03.230183 896 tablet_service.cc:1505] Processing CreateTablet for tablet d64b1a762f124195b4369c096587881d (DEFAULT_TABLE table=default.Table_859570febd594b1782eaa7388665eef0 [id=12d36151f38e4a5783f82884f1eea461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.230683 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64b1a762f124195b4369c096587881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.232132 784 raft_consensus.cc:3060] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:03.232955 895 tablet_service.cc:1505] Processing CreateTablet for tablet cf4934c3842a463e857262ceed37e0bc (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.233501 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4934c3842a463e857262ceed37e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.235749 784 raft_consensus.cc:2468] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:03.236217 861 leader_election.cc:304] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:03.236428 861 raft_consensus.cc:2686] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:03.237891 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--webserver_interface=127.0.104.62
--webserver_port=37745
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 08:52:03.239503 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453911c5a29f43e7a21fa2c47ef264e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.241297 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9cf2a8f2254a38a725bcd3c83e8509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.241984 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.242051 757 tablet_service.cc:1505] Processing CreateTablet for tablet 25b2e13184de4c25a5f14bc78d871bda (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.242511 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2e13184de4c25a5f14bc78d871bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.242424 761 tablet_service.cc:1505] Processing CreateTablet for tablet be476c9994ed4dc28f606955b16bc4de (DEFAULT_TABLE table=default.Table_221ab665c3fc433288528a7dcc29c8f8 [id=6c75c2c51a93469489dfd59bce7906c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.242817 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be476c9994ed4dc28f606955b16bc4de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.244386 900 tablet_service.cc:1505] Processing CreateTablet for tablet 904523f7164849139cddb1100e95b824 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.250402 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c46023036d34ee7be5bb30a14caa0e2" } modify_external_catalogs: true
I20251029 08:52:03.250617 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c46023036d34ee7be5bb30a14caa0e2" } modify_external_catalogs: true
I20251029 08:52:03.253304 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7966c1de42494873aa075b0d9185824d" } modify_external_catalogs: true
I20251029 08:52:03.253465 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7966c1de42494873aa075b0d9185824d" } modify_external_catalogs: true
I20251029 08:52:03.254698 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_985f7f5558144c058beb6d77045f1924"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.255244 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_985f7f5558144c058beb6d77045f1924 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.256418 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3f0099e12b704855a8f80562d4b246da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.256892 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f0099e12b704855a8f80562d4b246da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.258162 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_55cdb37e69164060ae6595fe21e9db73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.258644 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55cdb37e69164060ae6595fe21e9db73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.260262 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0e93e5470b774809972ce32757764331"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.260732 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e93e5470b774809972ce32757764331 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.264039 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4f65f8ca443e48a48d7ef4a474dc95b9
I20251029 08:52:03.266350 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_07a54a54b70a4f0aaea54ff93aa4bb65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.266855 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.270318 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904523f7164849139cddb1100e95b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.272852 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.274210 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.274762 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ba1332d26ee0440386a364dcbfabd776"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.275403 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba1332d26ee0440386a364dcbfabd776 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.247524 896 tablet_service.cc:1505] Processing CreateTablet for tablet 24ed199b700f4f92858c4edb95217d03 (DEFAULT_TABLE table=default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.276032 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ed199b700f4f92858c4edb95217d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.276543 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d1f17f82721d4c14ac66469353b2bcc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:03.276656 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1f00f2933c364d019c2b2d749778ad6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.277117 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f00f2933c364d019c2b2d749778ad6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:03.277180 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1f17f82721d4c14ac66469353b2bcc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.278815 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 51bb01c123c44fb69b2f174fca5d7a13 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.280385 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4a3272a7a0a419dadd2cad132e996fd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:03.281966 588 master_service.cc:954] ReplaceTablet: received request to replace tablet db9cf2a8f2254a38a725bcd3c83e8509 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.283403 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1e34f0b88fa4485094c008516dd01ab2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.283878 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e34f0b88fa4485094c008516dd01ab2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.267742 1177 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.225167 1193 leader_election.cc:290] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.285593 755 tablet_service.cc:1505] Processing CreateTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (DEFAULT_TABLE table=default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.245656 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd (DEFAULT_TABLE table=default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.269755 759 tablet_service.cc:1505] Processing CreateTablet for tablet 06eb710809754e7abd00a345e8ae591a (DEFAULT_TABLE table=default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.285888 758 tablet_service.cc:1505] Processing CreateTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 (DEFAULT_TABLE table=default.Table_69a6a8b8bfed4039a0833b799b013493 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.286594 1177 tablet_bootstrap.cc:654] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.287310 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_859570febd594b1782eaa7388665eef0" } new_table_name: "default.Table_ba49ffb4353b45279109e5a2a2800c1f" modify_external_catalogs: true
I20251029 08:52:03.288658 1214 raft_consensus.cc:493] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.289911 763 tablet_service.cc:1505] Processing CreateTablet for tablet aea79bd4e6844b7abac623824cabbeeb (DEFAULT_TABLE table=default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.290278 1214 raft_consensus.cc:515] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.291054 1214 leader_election.cc:290] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.293212 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet be476c9994ed4dc28f606955b16bc4de
I20251029 08:52:03.293912 1175 ts_tablet_manager.cc:1916] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.294085 897 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.296684 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1d1d5b2a9e4051bbcd25be1578ee73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.298440 751 tablet_service.cc:1552] Processing DeleteTablet for tablet de8f868095a642f6a5d70c30bb74105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.298686 750 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.299623 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.299655 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:03.300541 588 catalog_manager.cc:6719] ReplaceTablet: tablet db9cf2a8f2254a38a725bcd3c83e8509 deleted and replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db
I20251029 08:52:03.301141 897 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.301473 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3f0099e12b704855a8f80562d4b246da" } new_table_name: "default.Table_e08d8a7733e048299c27863d38207448" modify_external_catalogs: true
I20251029 08:52:03.303330 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 (DEFAULT_TABLE table=default.Table_d67ccc3d92a84bb8abec8560b542ca91 [id=6f5f02000b50491682d2ab710c4e1f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.303814 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2dd9d6d0854e03a8298625938065c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.307607 752 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.267956 898 tablet_service.cc:1505] Processing CreateTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 (DEFAULT_TABLE table=default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.310784 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72f64830e924fa48e2fa4cf81ccbba3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.313462 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d64b1a762f124195b4369c096587881d consensus state has no leader
I20251029 08:52:03.299127 753 tablet_service.cc:1505] Processing CreateTablet for tablet 6e2c32b84201470aa0dd697a5da12117 (DEFAULT_TABLE table=default.Table_97621cac05024570b4f5355e6ea12e2d [id=3a1661ecd47240a79cf75ebbcaf6acf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.313733 746 tablet_service.cc:1505] Processing CreateTablet for tablet d64b1a762f124195b4369c096587881d (DEFAULT_TABLE table=default.Table_859570febd594b1782eaa7388665eef0 [id=12d36151f38e4a5783f82884f1eea461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.314261 747 tablet_service.cc:1505] Processing CreateTablet for tablet 904523f7164849139cddb1100e95b824 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.315032 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e339c9ed26a48a2a9fe172201c5cc60. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.314640 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9cf61d6137544019bca25acf2431986 consensus state has no leader
I20251029 08:52:03.314816 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a7141f1ff12e4dbfae563a27088ad272" } new_table_name: "default.Table_ec9165a4204e4f209e0a28ea2e0c04fe" modify_external_catalogs: true
I20251029 08:52:03.317513 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f1c66f1bd2cd4518ac472eca739914f8" } new_table_name: "default.Table_4265f825f08f4af9b9f1192847406c8c" modify_external_catalogs: true
I20251029 08:52:03.318894 1238 leader_election.cc:290] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:03.319416 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 6426b5a4f5784b3a97475cb23199f916 (DEFAULT_TABLE table=default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.319854 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6426b5a4f5784b3a97475cb23199f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.321097 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8f868095a642f6a5d70c30bb74105c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:03.321776 861 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.326164 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 (DEFAULT_TABLE table=default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.326956 1041 tablet_service.cc:1505] Processing CreateTablet for tablet d64b1a762f124195b4369c096587881d (DEFAULT_TABLE table=default.Table_859570febd594b1782eaa7388665eef0 [id=12d36151f38e4a5783f82884f1eea461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.332486 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64b1a762f124195b4369c096587881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.332782 900 tablet_service.cc:1505] Processing CreateTablet for tablet a9cf61d6137544019bca25acf2431986 (DEFAULT_TABLE table=default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.333184 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf61d6137544019bca25acf2431986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.333410 1037 tablet_service.cc:1505] Processing CreateTablet for tablet cf4934c3842a463e857262ceed37e0bc (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.334990 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a2b37e45e444caa3f5117ae9a25304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.307523 754 tablet_service.cc:1505] Processing CreateTablet for tablet 14adc0f009d94287b92eda7b171c7c13 (DEFAULT_TABLE table=default.Table_f1c66f1bd2cd4518ac472eca739914f8 [id=9a65ac9ea0f448a38b2098fcefd707dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.335407 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4934c3842a463e857262ceed37e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.338603 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74
I20251029 08:52:03.339221 595 catalog_manager.cc:6719] ReplaceTablet: tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 deleted and replaced by tablet 32c7b299284f4a238778c4a5dbf85611
I20251029 08:52:03.339998 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 51bb01c123c44fb69b2f174fca5d7a13
I20251029 08:52:03.340579 594 catalog_manager.cc:6719] ReplaceTablet: tablet 51bb01c123c44fb69b2f174fca5d7a13 deleted and replaced by tablet c4e1b6b6359b437aa295b473fa42dce1
I20251029 08:52:03.308581 750 tablet_service.cc:1505] Processing CreateTablet for tablet 954c264e600a46f19db40dde5180787d (DEFAULT_TABLE table=default.Table_a7141f1ff12e4dbfae563a27088ad272 [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.309599 749 tablet_service.cc:1505] Processing CreateTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 (DEFAULT_TABLE table=default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.313580 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6426b5a4f5784b3a97475cb23199f916 (DEFAULT_TABLE table=default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.328109 752 tablet_service.cc:1505] Processing CreateTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 (DEFAULT_TABLE table=default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.342068 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8a4b96d3c748daa0f467768b2b7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.308288 751 tablet_service.cc:1505] Processing CreateTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 (DEFAULT_TABLE table=default.Table_d67ccc3d92a84bb8abec8560b542ca91 [id=6f5f02000b50491682d2ab710c4e1f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.346421 1179 ts_tablet_manager.cc:1929] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.346580 1179 log.cc:1199] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/24826ee22d284341ac8ad0258ccfe567
I20251029 08:52:03.346935 1179 ts_tablet_manager.cc:1950] T 24826ee22d284341ac8ad0258ccfe567 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.348219 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 with delete_type TABLET_DATA_DELETED (replaced by tablet 32c7b299284f4a238778c4a5dbf85611) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.348871 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 24ed199b700f4f92858c4edb95217d03 (DEFAULT_TABLE table=default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.349924 745 tablet_service.cc:1505] Processing CreateTablet for tablet cf4934c3842a463e857262ceed37e0bc (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.353025 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.354887 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64b1a762f124195b4369c096587881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.355945 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ed199b700f4f92858c4edb95217d03. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.356134 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 954c264e600a46f19db40dde5180787d consensus state has no leader
I20251029 08:52:03.357321 1177 tablet_bootstrap.cc:492] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.357484 1177 ts_tablet_manager.cc:1403] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.003s
I20251029 08:52:03.357580 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.357520 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_eedf056f49564746a9c2c87482f5db42" } modify_external_catalogs: true
I20251029 08:52:03.357726 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_eedf056f49564746a9c2c87482f5db42" } modify_external_catalogs: true
I20251029 08:52:03.358323 1177 raft_consensus.cc:359] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.358528 1177 raft_consensus.cc:385] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.358621 1177 raft_consensus.cc:740] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.358846 1177 consensus_queue.cc:260] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.359349 1177 ts_tablet_manager.cc:1434] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:03.359356 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ba1332d26ee0440386a364dcbfabd776" } modify_external_catalogs: true
I20251029 08:52:03.359540 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ba1332d26ee0440386a364dcbfabd776" } modify_external_catalogs: true
I20251029 08:52:03.359658 1177 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.363314 1177 tablet_bootstrap.cc:654] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.364207 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a7ee229f4d4b2fa6dd1f75b4aaff74" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:03.367825 901 tablet_service.cc:1552] Processing DeleteTablet for tablet b4a3272a7a0a419dadd2cad132e996fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.369122 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.371243 1175 ts_tablet_manager.cc:1929] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.371428 1175 log.cc:1199] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/de8f868095a642f6a5d70c30bb74105c
I20251029 08:52:03.371502 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d881df3a5f1043079e80ac59d710e3ac" } modify_external_catalogs: true
I20251029 08:52:03.371707 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d881df3a5f1043079e80ac59d710e3ac" } modify_external_catalogs: true
I20251029 08:52:03.371889 1175 ts_tablet_manager.cc:1950] T de8f868095a642f6a5d70c30bb74105c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:03.372195 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 904523f7164849139cddb1100e95b824 (DEFAULT_TABLE table=default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.372912 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 06c08679f1e746538bfa2c80d4fbdba3 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.372884 1042 tablet_service.cc:1505] Processing CreateTablet for tablet a9cf61d6137544019bca25acf2431986 (DEFAULT_TABLE table=default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.376261 901 tablet_service.cc:1505] Processing CreateTablet for tablet 933817bf2058464f837ba96921c4ed63 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:03.376364 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14adc0f009d94287b92eda7b171c7c13 consensus state has no leader
I20251029 08:52:03.376824 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933817bf2058464f837ba96921c4ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.379177 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904523f7164849139cddb1100e95b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.379724 1179 tablet_replica.cc:333] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.380059 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 with delete_type TABLET_DATA_DELETED (replaced by tablet 32c7b299284f4a238778c4a5dbf85611) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.380221 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf61d6137544019bca25acf2431986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.382642 1179 raft_consensus.cc:2243] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.382640 902 tablet_service.cc:1505] Processing CreateTablet for tablet 4b55ae84c8d441459a5931abda471f6c (DEFAULT_TABLE table=default.Table_0e93e5470b774809972ce32757764331 [id=fa54a881d58045fdb998132a18e1c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.383127 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b55ae84c8d441459a5931abda471f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.383334 1179 raft_consensus.cc:2272] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.383661 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904523f7164849139cddb1100e95b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.383942 1179 ts_tablet_manager.cc:1916] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.386693 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 24826ee22d284341ac8ad0258ccfe567 (table default.Table_ec5e17a240a046d78a1c65dde9e90397 [id=2e26f7325f4640479006670b114d0f59]) successfully deleted
I20251029 08:52:03.387831 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:03.389559 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 710e99b4116744289f9a2130f65f4b94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 710e99b4116744289f9a2130f65f4b94 already in progress: creating tablet
I20251029 08:52:03.391410 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954c264e600a46f19db40dde5180787d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.399147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2dd9d6d0854e03a8298625938065c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.402688 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a2b37e45e444caa3f5117ae9a25304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.409219 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 32c7b299284f4a238778c4a5dbf85611
I20251029 08:52:03.409228 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4934c3842a463e857262ceed37e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.410380 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14adc0f009d94287b92eda7b171c7c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.411846 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2c32b84201470aa0dd697a5da12117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.412436 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a7ee229f4d4b2fa6dd1f75b4aaff74" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:03.412946 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72f64830e924fa48e2fa4cf81ccbba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.413816 1264 tablet.cc:1722] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:03.414147 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6426b5a4f5784b3a97475cb23199f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.414633 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4e808bff7558472099587a0f95c002ff
I20251029 08:52:03.415611 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea79bd4e6844b7abac623824cabbeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.416103 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06eb710809754e7abd00a345e8ae591a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.416725 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6e2c32b84201470aa0dd697a5da12117
I20251029 08:52:03.416716 897 tablet_service.cc:1505] Processing CreateTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 (DEFAULT_TABLE table=default.Table_985f7f5558144c058beb6d77045f1924 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.417148 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b459b366cb402db2e7e8fee2b3e855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.417570 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.418145 904 tablet_service.cc:1505] Processing CreateTablet for tablet 4e808bff7558472099587a0f95c002ff (DEFAULT_TABLE table=default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.418645 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e808bff7558472099587a0f95c002ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.418973 718 leader_election.cc:343] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:03.422526 996 leader_election.cc:343] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:03.423631 998 leader_election.cc:343] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b4a3272a7a0a419dadd2cad132e996fd
I20251029 08:52:03.423951 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ba49ffb4353b45279109e5a2a2800c1f" } new_table_name: "default.Table_cb01810d231f447986917d7d5d1276b7" modify_external_catalogs: true
I20251029 08:52:03.424145 998 leader_election.cc:304] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.424444 1238 raft_consensus.cc:2749] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.424624 1183 tablet_bootstrap.cc:492] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.425048 1183 ts_tablet_manager.cc:1403] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.517s user 0.014s sys 0.015s
I20251029 08:52:03.426069 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.426220 1183 raft_consensus.cc:359] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.428596 762 tablet_service.cc:1505] Processing CreateTablet for tablet 24ed199b700f4f92858c4edb95217d03 (DEFAULT_TABLE table=default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.429057 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ed199b700f4f92858c4edb95217d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.429977 899 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b86f39c994ea5930a6a970830093f (DEFAULT_TABLE table=default.Table_1e34f0b88fa4485094c008516dd01ab2 [id=2f82075a67804ac48d6817dd2924b9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.430483 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b86f39c994ea5930a6a970830093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.432898 1177 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.433216 1177 ts_tablet_manager.cc:1403] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20251029 08:52:03.433499 1214 raft_consensus.cc:493] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.433660 1214 raft_consensus.cc:515] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.434329 1214 leader_election.cc:290] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.434300 1177 raft_consensus.cc:359] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.434774 1177 raft_consensus.cc:385] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.434865 1177 raft_consensus.cc:740] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.435297 1177 consensus_queue.cc:260] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.435546 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.435940 1177 ts_tablet_manager.cc:1434] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:03.437767 1177 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.438524 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 06c08679f1e746538bfa2c80d4fbdba3
I20251029 08:52:03.439128 593 catalog_manager.cc:6719] ReplaceTablet: tablet 06c08679f1e746538bfa2c80d4fbdba3 deleted and replaced by tablet a73a891fb6034f989d29a3f837303d84
I20251029 08:52:03.439545 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.439808 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4adb698e66ef4612946087c05b279db2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.440517 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4adb698e66ef4612946087c05b279db2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:03.451563 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d64b1a762f124195b4369c096587881d consensus state has no leader
I20251029 08:52:03.448401 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.452051 1044 tablet_service.cc:1505] Processing CreateTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 (DEFAULT_TABLE table=default.Table_985f7f5558144c058beb6d77045f1924 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.453166 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 4b55ae84c8d441459a5931abda471f6c (DEFAULT_TABLE table=default.Table_0e93e5470b774809972ce32757764331 [id=fa54a881d58045fdb998132a18e1c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.455149 1038 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b86f39c994ea5930a6a970830093f (DEFAULT_TABLE table=default.Table_1e34f0b88fa4485094c008516dd01ab2 [id=2f82075a67804ac48d6817dd2924b9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.457206 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 933817bf2058464f837ba96921c4ed63 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.457326 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.459267 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.459534 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b86f39c994ea5930a6a970830093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.460451 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b55ae84c8d441459a5931abda471f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.461521 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:03.461524 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]) successfully deleted
I20251029 08:52:03.462107 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933817bf2058464f837ba96921c4ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.462512 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:03.464221 900 tablet_service.cc:1505] Processing CreateTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.464740 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8cb0c6c6a1c4877a89476d4822f48db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.438112 755 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:03.445863 720 leader_election.cc:343] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.467527 720 leader_election.cc:304] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.458895 751 tablet_service.cc:1505] Processing CreateTablet for tablet a9cf61d6137544019bca25acf2431986 (DEFAULT_TABLE table=default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.467527 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.468128 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf61d6137544019bca25acf2431986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.468518 755 tablet_service.cc:1505] Processing CreateTablet for tablet 933817bf2058464f837ba96921c4ed63 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.468847 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933817bf2058464f837ba96921c4ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.469165 1214 raft_consensus.cc:493] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.470876 747 tablet_service.cc:1505] Processing CreateTablet for tablet 4b55ae84c8d441459a5931abda471f6c (DEFAULT_TABLE table=default.Table_0e93e5470b774809972ce32757764331 [id=fa54a881d58045fdb998132a18e1c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.472399 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.472958 750 tablet_service.cc:1505] Processing CreateTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 (DEFAULT_TABLE table=default.Table_985f7f5558144c058beb6d77045f1924 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:03.473009 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1d1d5b2a9e4051bbcd25be1578ee73 already in progress: creating tablet
I20251029 08:52:03.473227 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 728e64abaadc46598bf2dbc32dc55bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.473650 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b55ae84c8d441459a5931abda471f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.474609 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 with delete_type TABLET_DATA_DELETED (replaced by tablet 32c7b299284f4a238778c4a5dbf85611) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.475073 1051 tablet_service.cc:1505] Processing CreateTablet for tablet a61235da8c7a4f31b29d5d50c53c347d (DEFAULT_TABLE table=default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.475482 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61235da8c7a4f31b29d5d50c53c347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.475641 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8ad0b47dd01d4cf6ae72cee187fe7624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.476388 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ad0b47dd01d4cf6ae72cee187fe7624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.478260 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bc1f6a8467ac4e1084471956749c7145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.478819 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc1f6a8467ac4e1084471956749c7145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.480213 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_80b0f80eda77452b959592ce98b44953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.480752 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80b0f80eda77452b959592ce98b44953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.480671 1214 raft_consensus.cc:515] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.481314 1214 leader_election.cc:290] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.481506 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b459b366cb402db2e7e8fee2b3e855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.482273 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 9487a5ebf7ac404695ef85848278ec8d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.475050 1183 raft_consensus.cc:385] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.483296 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_70788044eecf4c7da477e61f7610e1c2" } modify_external_catalogs: true
I20251029 08:52:03.483566 1183 raft_consensus.cc:740] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.483564 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_70788044eecf4c7da477e61f7610e1c2" } modify_external_catalogs: true
I20251029 08:52:03.483855 1183 consensus_queue.cc:260] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:03.484627 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e899e183e255498b83a6d740af98de42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e899e183e255498b83a6d740af98de42 already in progress: creating tablet
I20251029 08:52:03.485039 1175 tablet_replica.cc:333] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:03.485010 758 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.486133 1214 raft_consensus.cc:2749] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.488283 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:03.488303 1178 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.123s sys 0.176s
W20251029 08:52:03.488425 1178 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.514s user 0.123s sys 0.176s
I20251029 08:52:03.490374 758 tablet_service.cc:1505] Processing CreateTablet for tablet 4e808bff7558472099587a0f95c002ff (DEFAULT_TABLE table=default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.493227 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ff2ff839cd5b4f62a2b81052c1c709b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.493854 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff2ff839cd5b4f62a2b81052c1c709b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.483264 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 4e808bff7558472099587a0f95c002ff (DEFAULT_TABLE table=default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.494541 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e808bff7558472099587a0f95c002ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.494647 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.487905 1048 tablet_service.cc:1505] Processing CreateTablet for tablet b273c488f33c498890f1739380b18239 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.496407 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b273c488f33c498890f1739380b18239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.503075 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e808bff7558472099587a0f95c002ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.503363 1183 ts_tablet_manager.cc:1434] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.078s user 0.001s sys 0.001s
I20251029 08:52:03.504451 1051 tablet_service.cc:1505] Processing CreateTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.505232 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_61b80d11cc21467faa27e4f5ce211cc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.505812 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b80d11cc21467faa27e4f5ce211cc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.507515 761 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b86f39c994ea5930a6a970830093f (DEFAULT_TABLE table=default.Table_1e34f0b88fa4485094c008516dd01ab2 [id=2f82075a67804ac48d6817dd2924b9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.508996 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66c8696f5f854516b099c64d58891cdb" } modify_external_catalogs: true
I20251029 08:52:03.509162 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66c8696f5f854516b099c64d58891cdb" } modify_external_catalogs: true
I20251029 08:52:03.509428 757 tablet_service.cc:1505] Processing CreateTablet for tablet a61235da8c7a4f31b29d5d50c53c347d (DEFAULT_TABLE table=default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.510176 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e4335ad852264f3382fd20c90284b5c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.510677 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4335ad852264f3382fd20c90284b5c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.511057 752 tablet_service.cc:1505] Processing CreateTablet for tablet b273c488f33c498890f1739380b18239 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.515134 1175 raft_consensus.cc:2243] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.516563 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61235da8c7a4f31b29d5d50c53c347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.517665 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b273c488f33c498890f1739380b18239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.518584 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b86f39c994ea5930a6a970830093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.522887 760 tablet_service.cc:1505] Processing CreateTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db (DEFAULT_TABLE table=default.Table_9df4bbf8684046b9876226a4a342654c [id=c99d60e34d3a4ba1b1a36319e7584543]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.526301 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b459b366cb402db2e7e8fee2b3e855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.527465 1178 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:03.527683 902 tablet_service.cc:1505] Processing CreateTablet for tablet b273c488f33c498890f1739380b18239 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.528631 1175 raft_consensus.cc:2272] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.529493 898 tablet_service.cc:1505] Processing CreateTablet for tablet a61235da8c7a4f31b29d5d50c53c347d (DEFAULT_TABLE table=default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.530161 1175 ts_tablet_manager.cc:1916] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.531492 1183 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.532575 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c9a18c9e63d4b5e9a42fe053b187b08" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:03.534857 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b273c488f33c498890f1739380b18239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.535362 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8cb0c6c6a1c4877a89476d4822f48db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.538908 1178 ts_tablet_manager.cc:1403] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.556s user 0.124s sys 0.177s
I20251029 08:52:03.544446 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61235da8c7a4f31b29d5d50c53c347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.534096 1261 tablet.cc:1722] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:03.549868 1175 ts_tablet_manager.cc:1929] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.557710 1175 log.cc:1199] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:03.559044 1175 ts_tablet_manager.cc:1950] T f7f8ae2eeabd419b8999372e6aa922ae P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:03.560698 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet f7f8ae2eeabd419b8999372e6aa922ae (table default.Table_2c76bf4e84c04dfca8b4897658618b06 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:03.558615 1177 tablet_bootstrap.cc:654] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.550088 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:03.538161 718 leader_election.cc:343] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:03.565402 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f65f8ca443e48a48d7ef4a474dc95b9 already in progress: creating tablet
W20251029 08:52:03.566092 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet be476c9994ed4dc28f606955b16bc4de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be476c9994ed4dc28f606955b16bc4de already in progress: creating tablet
I20251029 08:52:03.568214 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6fe27de4b7664cd5907bfb6507779a9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.569126 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fe27de4b7664cd5907bfb6507779a9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.570382 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_affbc133198140a2b050f47a2d51c45d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.570829 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_affbc133198140a2b050f47a2d51c45d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.571087 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2c76bf4e84c04dfca8b4897658618b06" } new_table_name: "default.Table_aa83a84a696b4573b7ceeb7a274a698e" modify_external_catalogs: true
I20251029 08:52:03.571252 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_35f72924484241fbaf61c2cafa158f4e" } new_table_name: "default.Table_03feaca359af4f749d12bfa146dc1859" modify_external_catalogs: true
W20251029 08:52:03.574553 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet db9cf2a8f2254a38a725bcd3c83e8509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9cf2a8f2254a38a725bcd3c83e8509 already in progress: creating tablet
I20251029 08:52:03.575817 1175 tablet_replica.cc:333] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:03.576789 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 44555.8us
W20251029 08:52:03.581343 718 leader_election.cc:343] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.583679 1175 raft_consensus.cc:2243] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.583926 1175 raft_consensus.cc:2272] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.587164 1175 ts_tablet_manager.cc:1916] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.589175 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8cb0c6c6a1c4877a89476d4822f48db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.593247 1183 tablet_bootstrap.cc:654] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.593650 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9f8030a3d166447fbd35405c27dc35ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.594331 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f8030a3d166447fbd35405c27dc35ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.607786 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6426b5a4f5784b3a97475cb23199f916
I20251029 08:52:03.608690 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.609635 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 728e64abaadc46598bf2dbc32dc55bdd from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.612452 1177 tablet_bootstrap.cc:492] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.614009 592 master_service.cc:954] ReplaceTablet: received request to replace tablet a709caf6b7244a2d8ab45f6f578cf7cf from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.616314 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4265f825f08f4af9b9f1192847406c8c" } new_table_name: "default.Table_0eecd4c045104f77bbeb969426549226" modify_external_catalogs: true
I20251029 08:52:03.620908 1177 ts_tablet_manager.cc:1403] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.183s user 0.004s sys 0.001s
I20251029 08:52:03.622676 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.626574 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.631760 1179 ts_tablet_manager.cc:1929] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.631695 1177 raft_consensus.cc:359] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.631886 1177 raft_consensus.cc:385] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.631986 1177 raft_consensus.cc:740] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.632411 1177 consensus_queue.cc:260] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.632910 1177 ts_tablet_manager.cc:1434] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251029 08:52:03.633272 1177 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.628209 1178 raft_consensus.cc:359] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.638285 1179 log.cc:1199] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/04b17cb6f63a4064873d0851f4b563cb
I20251029 08:52:03.638676 1179 ts_tablet_manager.cc:1950] T 04b17cb6f63a4064873d0851f4b563cb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.639418 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 38) took 1035 ms. Trace:
I20251029 08:52:03.641871 1178 raft_consensus.cc:385] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.639541 1179 rpcz_store.cc:276] 1029 08:52:02.604359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.604429 (+ 70us) service_pool.cc:225] Handling call
1029 08:52:03.354052 (+749623us) ts_tablet_manager.cc:1151] Deleting tablet 04b17cb6f63a4064873d0851f4b563cb
1029 08:52:03.354061 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.401381 (+ 47320us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.631626 (+230245us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.639381 (+ 7755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":749430}
I20251029 08:52:03.642571 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:03.644134 1183 tablet_bootstrap.cc:492] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.644354 1183 ts_tablet_manager.cc:1403] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.002s
I20251029 08:52:03.645015 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_517bd3e187ef44cb9a149605bbdcaebd" } new_table_name: "default.Table_53de717e75854922a7088385152a36af" modify_external_catalogs: true
I20251029 08:52:03.645177 1183 raft_consensus.cc:359] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.645414 1183 raft_consensus.cc:385] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.645494 1183 raft_consensus.cc:740] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.645767 1183 consensus_queue.cc:260] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.647878 1183 ts_tablet_manager.cc:1434] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:03.648382 1183 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.648710 1178 raft_consensus.cc:740] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.648983 1178 consensus_queue.cc:260] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.655449 1183 tablet_bootstrap.cc:654] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.655865 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 25b2e13184de4c25a5f14bc78d871bda from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.655683 1179 tablet_replica.cc:333] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.658027 1176 ts_tablet_manager.cc:1929] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.658668 1178 ts_tablet_manager.cc:1434] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.110s user 0.001s sys 0.001s
I20251029 08:52:03.659345 1179 raft_consensus.cc:2243] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.659523 1179 raft_consensus.cc:2272] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.659953 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 710e99b4116744289f9a2130f65f4b94 with delete_type TABLET_DATA_DELETED (replaced by tablet b4ceeaba655542dab2556d3988b6e841) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.660764 1176 log.cc:1199] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4b8068c1c7ce49fc85b660f40a7d1c58
I20251029 08:52:03.663712 1176 ts_tablet_manager.cc:1950] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.667834 1178 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:03.668428 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
W20251029 08:52:03.670094 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0d8a59f8f764466686a4b25c3b28596f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d8a59f8f764466686a4b25c3b28596f already in progress: creating tablet
I20251029 08:52:03.670722 1179 ts_tablet_manager.cc:1916] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:03.670786 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 196030us
I20251029 08:52:03.671604 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c9a18c9e63d4b5e9a42fe053b187b08" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:03.671978 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ad0b47dd01d4cf6ae72cee187fe7624" } modify_external_catalogs: true
I20251029 08:52:03.672211 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ad0b47dd01d4cf6ae72cee187fe7624" } modify_external_catalogs: true
I20251029 08:52:03.675498 900 tablet_service.cc:1552] Processing DeleteTablet for tablet aa73cfd89a724aada8af426a56b3481a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.677373 1177 tablet_bootstrap.cc:654] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:03.678392 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 57831.6us
I20251029 08:52:03.680811 1266 raft_consensus.cc:493] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.682686 1176 tablet_replica.cc:333] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.685401 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3a8b8f91860e419e8538f028e57603fd" } modify_external_catalogs: true
I20251029 08:52:03.685611 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3a8b8f91860e419e8538f028e57603fd" } modify_external_catalogs: true
I20251029 08:52:03.691331 1176 raft_consensus.cc:2243] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:03.692989 1183 tablet_bootstrap.cc:492] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.693272 1183 ts_tablet_manager.cc:1403] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.002s
I20251029 08:52:03.694087 1183 raft_consensus.cc:359] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.694370 1183 raft_consensus.cc:385] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.694481 1183 raft_consensus.cc:740] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.694813 1183 consensus_queue.cc:260] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:03.695300 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73a891fb6034f989d29a3f837303d84 has no consensus state
I20251029 08:52:03.697579 1183 ts_tablet_manager.cc:1434] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:03.698900 1183 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.700321 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_076b5e5e98044d93b9c0d75227b7e0c3" } modify_external_catalogs: true
I20251029 08:52:03.700546 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_076b5e5e98044d93b9c0d75227b7e0c3" } modify_external_catalogs: true
I20251029 08:52:03.702003 1176 raft_consensus.cc:2272] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.680980 1266 raft_consensus.cc:515] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.702986 1266 leader_election.cc:290] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.700644 1178 tablet_bootstrap.cc:654] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.703780 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a232a4a898f04f339084c807990a14e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:03.704897 718 leader_election.cc:343] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.705691 1176 ts_tablet_manager.cc:1916] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.706924 1175 ts_tablet_manager.cc:1929] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.707080 1175 log.cc:1199] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/b4a3272a7a0a419dadd2cad132e996fd
W20251029 08:52:03.710917 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a232a4a898f04f339084c807990a14e1 consensus state has no leader
I20251029 08:52:03.711536 1183 tablet_bootstrap.cc:654] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.713482 1179 ts_tablet_manager.cc:1929] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.713686 1179 log.cc:1199] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b4a3272a7a0a419dadd2cad132e996fd
I20251029 08:52:03.714092 1179 ts_tablet_manager.cc:1950] T b4a3272a7a0a419dadd2cad132e996fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.714928 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 39) took 1045 ms. Trace:
I20251029 08:52:03.715072 1179 rpcz_store.cc:276] 1029 08:52:02.669111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.669257 (+ 146us) service_pool.cc:225] Handling call
1029 08:52:03.655620 (+986363us) ts_tablet_manager.cc:1151] Deleting tablet b4a3272a7a0a419dadd2cad132e996fd
1029 08:52:03.655630 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.678358 (+ 22728us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.713368 (+ 35010us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.714878 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":986090}
I20251029 08:52:03.715394 1176 ts_tablet_manager.cc:1929] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:03.715581 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b4a3272a7a0a419dadd2cad132e996fd (table default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]) successfully deleted
I20251029 08:52:03.717342 1179 tablet_replica.cc:333] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.717559 1179 raft_consensus.cc:2243] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.717818 1179 raft_consensus.cc:2272] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.718386 1179 ts_tablet_manager.cc:1916] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.718979 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_61b80d11cc21467faa27e4f5ce211cc8" } new_table_name: "default.Table_d1d79c50455c4a05a0a804d863b09557" modify_external_catalogs: true
I20251029 08:52:03.722648 1183 tablet_bootstrap.cc:492] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.722839 1183 ts_tablet_manager.cc:1403] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20251029 08:52:03.723642 1183 raft_consensus.cc:359] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.723870 1183 raft_consensus.cc:385] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.723984 1183 raft_consensus.cc:740] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.724299 1183 consensus_queue.cc:260] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.726323 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e2c32b84201470aa0dd697a5da12117 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.728088 1183 ts_tablet_manager.cc:1434] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:03.730401 1176 log.cc:1199] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e9af06b4d79441968bda6bd7d67d76bf
I20251029 08:52:03.730235 1178 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:03.730702 1178 ts_tablet_manager.cc:1403] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.000s
I20251029 08:52:03.731212 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c4b692fd4fae4d93bd787b008adc3b20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:03.731534 1178 raft_consensus.cc:359] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.731729 1178 raft_consensus.cc:385] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:03.731779 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4b692fd4fae4d93bd787b008adc3b20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.731805 1178 raft_consensus.cc:740] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.732038 1178 consensus_queue.cc:260] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:03.720693 720 leader_election.cc:343] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.732333 720 leader_election.cc:304] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.735044 1183 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.736093 1266 raft_consensus.cc:2749] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:03.736438 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 64031.6us
W20251029 08:52:03.736721 720 leader_election.cc:343] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.736780 1183 tablet_bootstrap.cc:654] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.736897 720 leader_election.cc:304] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.737200 1266 raft_consensus.cc:2749] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.737944 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a232a4a898f04f339084c807990a14e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:03.738538 720 leader_election.cc:343] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.739096 720 leader_election.cc:304] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.739398 1266 raft_consensus.cc:2749] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.738996 1178 ts_tablet_manager.cc:1434] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 08:52:03.742437 1178 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:03.743155 593 master_service.cc:954] ReplaceTablet: received request to replace tablet b4ceeaba655542dab2556d3988b6e841 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.744107 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 9487a5ebf7ac404695ef85848278ec8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.745513 1175 ts_tablet_manager.cc:1950] T b4a3272a7a0a419dadd2cad132e996fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:03.745870 1176 ts_tablet_manager.cc:1950] T e9af06b4d79441968bda6bd7d67d76bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.746307 1179 ts_tablet_manager.cc:1929] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.746650 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet b4a3272a7a0a419dadd2cad132e996fd (table default.Table_1c1bec248bbd44408a382604c06f75f1 [id=7792b0ecd74d4b08bb0247c46eb0608d]) successfully deleted
I20251029 08:52:03.747568 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_0567a7bfaea44a6d8bd8807e6206939c [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:03.746572 1178 tablet_bootstrap.cc:654] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.750424 1183 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.750655 1183 ts_tablet_manager.cc:1403] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
W20251029 08:52:03.751360 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a709caf6b7244a2d8ab45f6f578cf7cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a709caf6b7244a2d8ab45f6f578cf7cf already in progress: creating tablet
I20251029 08:52:03.751421 1183 raft_consensus.cc:359] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:03.751729 1183 raft_consensus.cc:385] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.751837 1183 raft_consensus.cc:740] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.752147 1183 consensus_queue.cc:260] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:03.752666 1176 tablet_replica.cc:333] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.753705 1176 raft_consensus.cc:2243] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.755013 1183 ts_tablet_manager.cc:1434] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 08:52:03.756168 1183 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.758354 1183 tablet_bootstrap.cc:654] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:03.760280 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6b8a4b96d3c748daa0f467768b2b7abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b8a4b96d3c748daa0f467768b2b7abd already in progress: creating tablet
W20251029 08:52:03.760540 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8ded7b3943e544e585fe52433ff7555f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ded7b3943e544e585fe52433ff7555f already in progress: creating tablet
W20251029 08:52:03.760713 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1d1d5b2a9e4051bbcd25be1578ee73 already in progress: creating tablet
I20251029 08:52:03.763304 902 tablet_service.cc:1505] Processing CreateTablet for tablet a73a891fb6034f989d29a3f837303d84 (DEFAULT_TABLE table=default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.758531 1179 log.cc:1199] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aa73cfd89a724aada8af426a56b3481a
I20251029 08:52:03.759603 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e4cdf03acb92449d847aec897868b967 (DEFAULT_TABLE table=default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.763741 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73a891fb6034f989d29a3f837303d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.763861 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cdf03acb92449d847aec897868b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.759940 1048 tablet_service.cc:1505] Processing CreateTablet for tablet a73a891fb6034f989d29a3f837303d84 (DEFAULT_TABLE table=default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.764278 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73a891fb6034f989d29a3f837303d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.766016 1176 raft_consensus.cc:2272] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.767501 1176 ts_tablet_manager.cc:1916] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.760275 1043 tablet_service.cc:1505] Processing CreateTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.768327 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e1b6b6359b437aa295b473fa42dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.768249 760 tablet_service.cc:1505] Processing CreateTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed (DEFAULT_TABLE table=default.Table_d1f17f82721d4c14ac66469353b2bcc9 [id=63588e39919a481c856255fd848bbf98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.768392 898 tablet_service.cc:1505] Processing CreateTablet for tablet e4cdf03acb92449d847aec897868b967 (DEFAULT_TABLE table=default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.768857 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cdf03acb92449d847aec897868b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.761276 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed (DEFAULT_TABLE table=default.Table_d1f17f82721d4c14ac66469353b2bcc9 [id=63588e39919a481c856255fd848bbf98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.762792 1238 raft_consensus.cc:493] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.775678 1177 tablet_bootstrap.cc:492] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.775929 1177 ts_tablet_manager.cc:1403] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.143s user 0.004s sys 0.000s
I20251029 08:52:03.776790 1177 raft_consensus.cc:359] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:03.776942 1238 raft_consensus.cc:515] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.777062 1177 raft_consensus.cc:385] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.777153 1177 raft_consensus.cc:740] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.777421 1177 consensus_queue.cc:260] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:03.777958 1177 ts_tablet_manager.cc:1434] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:03.778085 1238 leader_election.cc:290] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:03.778326 1177 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:03.779127 996 leader_election.cc:343] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:03.779587 762 tablet_service.cc:1505] Processing CreateTablet for tablet e4cdf03acb92449d847aec897868b967 (DEFAULT_TABLE table=default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.779820 749 tablet_service.cc:1505] Processing CreateTablet for tablet a73a891fb6034f989d29a3f837303d84 (DEFAULT_TABLE table=default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.780141 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cdf03acb92449d847aec897868b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.780784 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73a891fb6034f989d29a3f837303d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.780934 900 tablet_service.cc:1505] Processing CreateTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed (DEFAULT_TABLE table=default.Table_d1f17f82721d4c14ac66469353b2bcc9 [id=63588e39919a481c856255fd848bbf98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.781275 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a756d6b5b3648e88cd2bd3400e51bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.782976 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a756d6b5b3648e88cd2bd3400e51bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.785409 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a756d6b5b3648e88cd2bd3400e51bed. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:03.786329 998 leader_election.cc:343] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:03.786345 747 tablet_service.cc:1505] Processing CreateTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.786700 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e1b6b6359b437aa295b473fa42dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.786708 998 leader_election.cc:304] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.787099 1238 raft_consensus.cc:2749] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.787330 902 tablet_service.cc:1505] Processing CreateTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 (DEFAULT_TABLE table=default.Table_517bd3e187ef44cb9a149605bbdcaebd [id=260ff4d973084d2b9572c2f4703c9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:03.787789 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e1b6b6359b437aa295b473fa42dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:03.788482 1178 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:03.788635 1178 ts_tablet_manager.cc:1403] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20251029 08:52:03.794198 1179 ts_tablet_manager.cc:1950] T aa73cfd89a724aada8af426a56b3481a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.795037 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 40) took 1124 ms. Trace:
W20251029 08:52:03.797034 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6e2c32b84201470aa0dd697a5da12117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2c32b84201470aa0dd697a5da12117 already in progress: creating tablet
W20251029 08:52:03.797672 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4e808bff7558472099587a0f95c002ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e808bff7558472099587a0f95c002ff already in progress: creating tablet
I20251029 08:52:03.797906 1179 rpcz_store.cc:276] 1029 08:52:02.670630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.672397 (+ 1767us) service_pool.cc:225] Handling call
1029 08:52:03.717269 (+1044872us) ts_tablet_manager.cc:1151] Deleting tablet aa73cfd89a724aada8af426a56b3481a
1029 08:52:03.717278 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.740974 (+ 23696us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.746155 (+ 5181us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.795003 (+ 48848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044673}
W20251029 08:52:03.798355 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 06c08679f1e746538bfa2c80d4fbdba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c08679f1e746538bfa2c80d4fbdba3 already in progress: creating tablet
I20251029 08:52:03.798424 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aa73cfd89a724aada8af426a56b3481a (table default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]) successfully deleted
I20251029 08:52:03.798833 1179 tablet_replica.cc:333] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:03.802112 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14adc0f009d94287b92eda7b171c7c13 consensus state has no leader
I20251029 08:52:03.802436 1179 raft_consensus.cc:2243] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.802829 1179 raft_consensus.cc:2272] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.803558 1179 ts_tablet_manager.cc:1916] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.805156 1178 raft_consensus.cc:359] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.805425 1178 raft_consensus.cc:385] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.805629 1178 raft_consensus.cc:740] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.805953 1175 tablet_replica.cc:333] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:03.805959 1178 consensus_queue.cc:260] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.806149 1175 raft_consensus.cc:2243] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.806334 1175 raft_consensus.cc:2272] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.810972 1178 ts_tablet_manager.cc:1434] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251029 08:52:03.812470 1178 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:03.822568 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_29ca0b751907426496780bf44111f5d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.823221 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29ca0b751907426496780bf44111f5d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.829777 1175 ts_tablet_manager.cc:1916] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.830193 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b8068c1c7ce49fc85b660f40a7d1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet 453911c5a29f43e7a21fa2c47ef264e8) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.819936 758 tablet_service.cc:1552] Processing DeleteTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:03.835912 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4e1b6b6359b437aa295b473fa42dce1 consensus state has no leader
W20251029 08:52:03.836782 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a232a4a898f04f339084c807990a14e1 (table default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a232a4a898f04f339084c807990a14e1 consensus state has no leader
I20251029 08:52:03.841269 1179 ts_tablet_manager.cc:1929] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.841754 1179 log.cc:1199] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/49cb49df9bee421188997cde00967852
I20251029 08:52:03.842367 1179 ts_tablet_manager.cc:1950] T 49cb49df9bee421188997cde00967852 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.844138 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 41) took 1171 ms. Trace:
I20251029 08:52:03.844354 1179 rpcz_store.cc:276] 1029 08:52:02.672794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.672998 (+ 204us) service_pool.cc:225] Handling call
1029 08:52:03.798763 (+1125765us) ts_tablet_manager.cc:1151] Deleting tablet 49cb49df9bee421188997cde00967852
1029 08:52:03.798775 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.808628 (+ 9853us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.811980 (+ 3352us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.844082 (+ 32102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125531}
I20251029 08:52:03.829993 1176 ts_tablet_manager.cc:1929] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.844871 1176 log.cc:1199] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/de8f868095a642f6a5d70c30bb74105c
I20251029 08:52:03.845357 1176 ts_tablet_manager.cc:1950] T de8f868095a642f6a5d70c30bb74105c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.846527 1238 raft_consensus.cc:493] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.846546 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]) successfully deleted
I20251029 08:52:03.846673 1238 raft_consensus.cc:515] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.846823 1176 tablet_replica.cc:333] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.847008 1176 raft_consensus.cc:2243] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:03.847048 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e899e183e255498b83a6d740af98de42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e899e183e255498b83a6d740af98de42 already in progress: creating tablet
I20251029 08:52:03.847193 1176 raft_consensus.cc:2272] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:03.847293 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e72f64830e924fa48e2fa4cf81ccbba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e72f64830e924fa48e2fa4cf81ccbba3 already in progress: creating tablet
I20251029 08:52:03.847347 1238 leader_election.cc:290] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:03.847878 1178 tablet_bootstrap.cc:654] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.848601 1176 ts_tablet_manager.cc:1916] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:03.848738 998 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: de8f868095a642f6a5d70c30bb74105c
W20251029 08:52:03.849176 996 leader_election.cc:343] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: de8f868095a642f6a5d70c30bb74105c
I20251029 08:52:03.849320 996 leader_election.cc:304] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.849478 1177 tablet_bootstrap.cc:654] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.849814 1238 raft_consensus.cc:2749] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.850567 1183 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.850746 1183 ts_tablet_manager.cc:1403] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.000s
I20251029 08:52:03.851529 1183 raft_consensus.cc:359] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.851773 1183 raft_consensus.cc:385] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.851909 1183 raft_consensus.cc:740] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.852200 1183 consensus_queue.cc:260] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.852447 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.853045 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.854277 1183 ts_tablet_manager.cc:1434] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:03.854357 1179 tablet_replica.cc:333] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.859010 1183 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.860986 1183 tablet_bootstrap.cc:654] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.862428 1176 ts_tablet_manager.cc:1929] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.862634 1176 log.cc:1199] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4f65f8ca443e48a48d7ef4a474dc95b9
I20251029 08:52:03.863083 1176 ts_tablet_manager.cc:1950] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.863692 589 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e71a09a3baa34324b2edf41468b063a8
I20251029 08:52:03.865744 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (table default.Table_7966c1de42494873aa075b0d9185824d [id=9b51cfa4d70542a280cc72c9528c45d4]) successfully deleted
I20251029 08:52:03.861758 1179 raft_consensus.cc:2243] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.866894 1179 raft_consensus.cc:2272] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.867677 1179 ts_tablet_manager.cc:1916] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:03.868537 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet be476c9994ed4dc28f606955b16bc4de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be476c9994ed4dc28f606955b16bc4de already in progress: creating tablet
I20251029 08:52:03.869192 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4b21589e83794d8e981ca8deeaf26b9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:03.869764 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b21589e83794d8e981ca8deeaf26b9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.870625 1176 tablet_replica.cc:333] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:03.871732 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 51bb01c123c44fb69b2f174fca5d7a13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51bb01c123c44fb69b2f174fca5d7a13 already in progress: creating tablet
I20251029 08:52:03.871732 1176 raft_consensus.cc:2243] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.871938 1176 raft_consensus.cc:2272] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:03.871999 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4e808bff7558472099587a0f95c002ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e808bff7558472099587a0f95c002ff already in progress: creating tablet
I20251029 08:52:03.872553 1176 ts_tablet_manager.cc:1916] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.877523 1183 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.877761 1183 ts_tablet_manager.cc:1403] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251029 08:52:03.878556 1183 raft_consensus.cc:359] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.878832 1183 raft_consensus.cc:385] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.878934 1183 raft_consensus.cc:740] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.879253 1183 consensus_queue.cc:260] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.880648 1177 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.880851 1177 ts_tablet_manager.cc:1403] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.001s
I20251029 08:52:03.881639 1183 ts_tablet_manager.cc:1434] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 08:52:03.881656 1177 raft_consensus.cc:359] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.881834 1177 raft_consensus.cc:385] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.881937 1177 raft_consensus.cc:740] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.882189 1177 consensus_queue.cc:260] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.883718 1183 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.887878 1183 tablet_bootstrap.cc:654] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:03.888599 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f has no consensus state
I20251029 08:52:03.889709 1177 ts_tablet_manager.cc:1434] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:03.891211 1176 ts_tablet_manager.cc:1929] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.891546 1176 log.cc:1199] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/07a7ee229f4d4b2fa6dd1f75b4aaff74
I20251029 08:52:03.891964 1176 ts_tablet_manager.cc:1950] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:03.893658 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet db9cf2a8f2254a38a725bcd3c83e8509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9cf2a8f2254a38a725bcd3c83e8509 already in progress: creating tablet
W20251029 08:52:03.894306 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6e2c32b84201470aa0dd697a5da12117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2c32b84201470aa0dd697a5da12117 already in progress: creating tablet
I20251029 08:52:03.890197 1177 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:03.894919 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 06c08679f1e746538bfa2c80d4fbdba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c08679f1e746538bfa2c80d4fbdba3 already in progress: creating tablet
W20251029 08:52:03.895547 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6426b5a4f5784b3a97475cb23199f916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6426b5a4f5784b3a97475cb23199f916 already in progress: creating tablet
I20251029 08:52:03.895875 1176 tablet_replica.cc:333] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.896150 1176 raft_consensus.cc:2243] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.896349 1177 tablet_bootstrap.cc:654] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.896430 1176 raft_consensus.cc:2272] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.897246 1176 ts_tablet_manager.cc:1916] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.897863 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7a0da910696e45fd9559acf90e71cde8" } modify_external_catalogs: true
I20251029 08:52:03.898046 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7a0da910696e45fd9559acf90e71cde8" } modify_external_catalogs: true
I20251029 08:52:03.898124 1179 ts_tablet_manager.cc:1929] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.898423 1179 log.cc:1199] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/728e64abaadc46598bf2dbc32dc55bdd
I20251029 08:52:03.899237 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 9487a5ebf7ac404695ef85848278ec8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.900513 1179 ts_tablet_manager.cc:1950] T 728e64abaadc46598bf2dbc32dc55bdd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.901502 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 42) took 1228 ms. Trace:
I20251029 08:52:03.901680 1179 rpcz_store.cc:276] 1029 08:52:02.673419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.673844 (+ 425us) service_pool.cc:225] Handling call
1029 08:52:03.846297 (+1172453us) ts_tablet_manager.cc:1151] Deleting tablet 728e64abaadc46598bf2dbc32dc55bdd
1029 08:52:03.846306 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.876919 (+ 30613us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.897996 (+ 21077us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.901463 (+ 3467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172274}
I20251029 08:52:03.902237 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 728e64abaadc46598bf2dbc32dc55bdd (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
I20251029 08:52:03.902607 1179 tablet_replica.cc:333] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.902838 1179 raft_consensus.cc:2243] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.903093 1179 raft_consensus.cc:2272] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.904259 1179 ts_tablet_manager.cc:1916] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.904932 1178 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:03.905164 1178 ts_tablet_manager.cc:1403] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20251029 08:52:03.906026 1178 raft_consensus.cc:359] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.906251 1176 ts_tablet_manager.cc:1929] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.907128 1176 log.cc:1199] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9487a5ebf7ac404695ef85848278ec8d
I20251029 08:52:03.906293 1178 raft_consensus.cc:385] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.907582 1176 ts_tablet_manager.cc:1950] T 9487a5ebf7ac404695ef85848278ec8d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.907629 1178 raft_consensus.cc:740] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.907814 1183 tablet_bootstrap.cc:492] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.908006 1183 ts_tablet_manager.cc:1403] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20251029 08:52:03.907969 1178 consensus_queue.cc:260] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.909026 1183 raft_consensus.cc:359] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.909273 1183 raft_consensus.cc:385] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.909400 1183 raft_consensus.cc:740] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.909459 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9487a5ebf7ac404695ef85848278ec8d (table default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]) successfully deleted
I20251029 08:52:03.909865 1176 tablet_replica.cc:333] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.909803 1183 consensus_queue.cc:260] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.910091 1176 raft_consensus.cc:2243] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.910301 1176 raft_consensus.cc:2272] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.911056 1176 ts_tablet_manager.cc:1916] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.911710 1178 ts_tablet_manager.cc:1434] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:03.912451 1183 ts_tablet_manager.cc:1434] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 08:52:03.913597 1183 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.915241 1183 tablet_bootstrap.cc:654] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.915772 1178 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:03.916949 1177 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.917109 1177 ts_tablet_manager.cc:1403] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20251029 08:52:03.917858 1177 raft_consensus.cc:359] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.918041 1177 raft_consensus.cc:385] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.918108 1177 raft_consensus.cc:740] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.918378 1177 consensus_queue.cc:260] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.918814 1177 ts_tablet_manager.cc:1434] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:03.919440 1176 ts_tablet_manager.cc:1929] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.920238 1176 log.cc:1199] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a709caf6b7244a2d8ab45f6f578cf7cf
I20251029 08:52:03.920661 1176 ts_tablet_manager.cc:1950] T a709caf6b7244a2d8ab45f6f578cf7cf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.920817 1193 raft_consensus.cc:493] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.921052 1177 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.920971 1193 raft_consensus.cc:515] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.921602 1193 leader_election.cc:290] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:03.922091 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a709caf6b7244a2d8ab45f6f578cf7cf (table default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]) successfully deleted
I20251029 08:52:03.922786 1177 tablet_bootstrap.cc:654] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:03.923272 861 leader_election.cc:343] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa73cfd89a724aada8af426a56b3481a
W20251029 08:52:03.923509 861 leader_election.cc:343] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: aa73cfd89a724aada8af426a56b3481a
I20251029 08:52:03.923620 861 leader_election.cc:304] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:03.924077 1193 raft_consensus.cc:2749] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:03.927938 1178 tablet_bootstrap.cc:654] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.936523 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 24ed199b700f4f92858c4edb95217d03
I20251029 08:52:03.939327 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 06eb710809754e7abd00a345e8ae591a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.940274 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.940680 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.941295 1176 tablet_replica.cc:333] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:03.941489 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.941550 1176 raft_consensus.cc:2243] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.941754 1176 raft_consensus.cc:2272] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.942553 1176 ts_tablet_manager.cc:1916] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.942665 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.945724 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 904523f7164849139cddb1100e95b824
I20251029 08:52:03.946123 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:03.946679 591 catalog_manager.cc:6719] ReplaceTablet: tablet 25b2e13184de4c25a5f14bc78d871bda deleted and replaced by tablet 983a406aeed243f29ab322f94a0f5239
I20251029 08:52:03.947423 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.947681 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.948261 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.948544 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.949020 1179 ts_tablet_manager.cc:1929] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.949249 1179 log.cc:1199] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/f7f8ae2eeabd419b8999372e6aa922ae
I20251029 08:52:03.949484 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.949774 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.949843 1179 ts_tablet_manager.cc:1950] T f7f8ae2eeabd419b8999372e6aa922ae P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:03.950579 1177 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.950670 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 43) took 1258 ms. Trace:
I20251029 08:52:03.950779 1177 ts_tablet_manager.cc:1403] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
I20251029 08:52:03.951843 1177 raft_consensus.cc:359] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.952103 1177 raft_consensus.cc:385] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.952237 1177 raft_consensus.cc:740] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.952524 1177 consensus_queue.cc:260] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.952468 1179 rpcz_store.cc:276] 1029 08:52:02.691948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.692010 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:03.902541 (+1210531us) ts_tablet_manager.cc:1151] Deleting tablet f7f8ae2eeabd419b8999372e6aa922ae
1029 08:52:03.902552 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.926689 (+ 24137us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.933727 (+ 7038us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:03.950616 (+ 16889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210263}
I20251029 08:52:03.953785 1177 ts_tablet_manager.cc:1434] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:03.954137 1177 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.954118 596 master_service.cc:954] ReplaceTablet: received request to replace tablet cf4934c3842a463e857262ceed37e0bc from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:03.954239 1176 ts_tablet_manager.cc:1929] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.954784 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_14e650112f054d8ea69743954a31706a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:03.955014 1179 tablet_replica.cc:333] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:03.955197 1179 raft_consensus.cc:2243] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:03.955358 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14e650112f054d8ea69743954a31706a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:03.955381 1179 raft_consensus.cc:2272] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.954808 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet f7f8ae2eeabd419b8999372e6aa922ae (table default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:03.956828 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 with delete_type TABLET_DATA_DELETED (replaced by tablet 32c7b299284f4a238778c4a5dbf85611) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.956871 1177 tablet_bootstrap.cc:654] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.960117 1175 ts_tablet_manager.cc:1929] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:03.962157 1175 log.cc:1199] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/728e64abaadc46598bf2dbc32dc55bdd
I20251029 08:52:03.962736 1175 ts_tablet_manager.cc:1950] T 728e64abaadc46598bf2dbc32dc55bdd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:03.962951 1176 log.cc:1199] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8ded7b3943e544e585fe52433ff7555f
I20251029 08:52:03.963768 1178 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:03.963948 1178 ts_tablet_manager.cc:1403] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20251029 08:52:03.964037 1179 ts_tablet_manager.cc:1916] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:03.964102 1183 tablet_bootstrap.cc:492] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:03.964864 1183 ts_tablet_manager.cc:1403] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20251029 08:52:03.964941 1178 raft_consensus.cc:359] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.965132 1178 raft_consensus.cc:385] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.965216 1178 raft_consensus.cc:740] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.965503 1178 consensus_queue.cc:260] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.965894 1183 raft_consensus.cc:359] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.966147 1183 raft_consensus.cc:385] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.966238 1183 raft_consensus.cc:740] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.966508 1183 consensus_queue.cc:260] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.967098 1183 ts_tablet_manager.cc:1434] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:03.967393 1178 ts_tablet_manager.cc:1434] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:03.967689 1178 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:03.969512 1178 tablet_bootstrap.cc:654] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.971712 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 728e64abaadc46598bf2dbc32dc55bdd (table default.Table_5a03fa86d2bc462a92a5d3e289c6a669 [id=5a1e4658e7f14b058f164483db5f84bb]) successfully deleted
W20251029 08:52:03.972114 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 51bb01c123c44fb69b2f174fca5d7a13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51bb01c123c44fb69b2f174fca5d7a13 already in progress: creating tablet
I20251029 08:52:03.973232 1183 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:03.973258 1177 tablet_bootstrap.cc:492] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.973404 1177 ts_tablet_manager.cc:1403] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251029 08:52:03.969877 1176 ts_tablet_manager.cc:1950] T 8ded7b3943e544e585fe52433ff7555f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:03.974165 1177 raft_consensus.cc:359] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.974397 1177 raft_consensus.cc:385] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.974486 1177 raft_consensus.cc:740] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.974633 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8ded7b3943e544e585fe52433ff7555f (table default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3 [id=5124fdf717764856bff708bedd9656c8]) successfully deleted
I20251029 08:52:03.974721 1177 consensus_queue.cc:260] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.975186 1177 ts_tablet_manager.cc:1434] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:03.975502 1177 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.977259 1238 raft_consensus.cc:493] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:03.977404 1238 raft_consensus.cc:515] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.978046 1238 leader_election.cc:290] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:03.978309 1177 tablet_bootstrap.cc:654] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:03.978284 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 24ed199b700f4f92858c4edb95217d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ed199b700f4f92858c4edb95217d03 already in progress: creating tablet
W20251029 08:52:03.979101 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6426b5a4f5784b3a97475cb23199f916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6426b5a4f5784b3a97475cb23199f916 already in progress: creating tablet
I20251029 08:52:03.979735 1183 tablet_bootstrap.cc:654] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.980576 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:03.981055 996 leader_election.cc:343] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.981345 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:03.981570 784 raft_consensus.cc:2468] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:03.981665 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 904523f7164849139cddb1100e95b824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904523f7164849139cddb1100e95b824 already in progress: creating tablet
I20251029 08:52:03.982098 1175 tablet_replica.cc:333] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:03.982340 1175 raft_consensus.cc:2243] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:03.982403 998 leader_election.cc:304] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:03.982870 1238 raft_consensus.cc:2804] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 08:52:03.982883 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 25b2e13184de4c25a5f14bc78d871bda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25b2e13184de4c25a5f14bc78d871bda already in progress: creating tablet
I20251029 08:52:03.982977 1238 raft_consensus.cc:493] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:03.983050 1238 raft_consensus.cc:3060] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:03.983148 1175 raft_consensus.cc:2272] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:03.986101 1238 raft_consensus.cc:515] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.987293 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:03.987879 996 leader_election.cc:343] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:03.989811 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:03.990406 593 catalog_manager.cc:6719] ReplaceTablet: tablet b4ceeaba655542dab2556d3988b6e841 deleted and replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:03.990746 902 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:03.991937 1238 leader_election.cc:290] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:03.992010 1177 tablet_bootstrap.cc:492] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:03.992165 1177 ts_tablet_manager.cc:1403] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20251029 08:52:03.992300 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1557722a7c8041c88ee5321c0944e234" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:03.992494 784 raft_consensus.cc:3060] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:03.992993 760 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:03.993008 1177 raft_consensus.cc:359] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.993216 1177 raft_consensus.cc:385] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:03.993336 1177 raft_consensus.cc:740] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:03.993633 1177 consensus_queue.cc:260] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:03.993813 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:03.994226 1177 ts_tablet_manager.cc:1434] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:03.995381 784 raft_consensus.cc:2468] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:03.995754 1177 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:03.996671 998 leader_election.cc:304] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:03.996748 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b4ceeaba655542dab2556d3988b6e841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ceeaba655542dab2556d3988b6e841 already in progress: creating tablet
I20251029 08:52:03.997392 1238 raft_consensus.cc:2804] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:03.997777 1177 tablet_bootstrap.cc:654] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:03.997848 1175 ts_tablet_manager.cc:1916] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.003966 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cf4934c3842a463e857262ceed37e0bc
I20251029 08:52:04.004458 596 catalog_manager.cc:6719] ReplaceTablet: tablet cf4934c3842a463e857262ceed37e0bc deleted and replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:04.005163 902 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.006168 760 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.006191 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:04.006822 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cf4934c3842a463e857262ceed37e0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf4934c3842a463e857262ceed37e0bc already in progress: creating tablet
I20251029 08:52:04.009788 1179 ts_tablet_manager.cc:1929] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.010066 1175 ts_tablet_manager.cc:1929] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.010097 1179 log.cc:1199] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9487a5ebf7ac404695ef85848278ec8d
I20251029 08:52:04.010433 1179 ts_tablet_manager.cc:1950] T 9487a5ebf7ac404695ef85848278ec8d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.011240 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 49) took 1121 ms. Trace:
I20251029 08:52:04.011417 1179 rpcz_store.cc:276] 1029 08:52:02.889717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.891445 (+ 1728us) service_pool.cc:225] Handling call
1029 08:52:03.954925 (+1063480us) ts_tablet_manager.cc:1151] Deleting tablet 9487a5ebf7ac404695ef85848278ec8d
1029 08:52:03.954938 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:03.987698 (+ 32760us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.009648 (+ 21950us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.011203 (+ 1555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054343}
I20251029 08:52:04.011917 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9487a5ebf7ac404695ef85848278ec8d (table default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]) successfully deleted
I20251029 08:52:04.012108 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 53) took 1122 ms. Trace:
I20251029 08:52:04.012275 1179 rpcz_store.cc:276] 1029 08:52:02.890080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.891517 (+ 1437us) service_pool.cc:225] Handling call
1029 08:52:04.011934 (+1120417us) ts_tablet_manager.cc:1151] Deleting tablet 0d8a59f8f764466686a4b25c3b28596f
1029 08:52:04.011946 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.012068 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103460}
W20251029 08:52:04.013108 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0d8a59f8f764466686a4b25c3b28596f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d8a59f8f764466686a4b25c3b28596f already in progress: creating tablet
I20251029 08:52:04.013195 1179 tablet_replica.cc:333] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.013418 1179 raft_consensus.cc:2243] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.013938 1179 raft_consensus.cc:2272] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.015027 1179 ts_tablet_manager.cc:1916] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.020080 1175 log.cc:1199] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aa73cfd89a724aada8af426a56b3481a
I20251029 08:52:04.020845 1175 ts_tablet_manager.cc:1950] T aa73cfd89a724aada8af426a56b3481a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.021328 1238 raft_consensus.cc:697] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:04.021847 1238 consensus_queue.cc:237] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.022586 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet aa73cfd89a724aada8af426a56b3481a (table default.Table_20e14b5aff1e4ecc878dc58302b78794 [id=2a44beb0e8e845e6bf116c77742d7dac]) successfully deleted
I20251029 08:52:04.023325 1179 ts_tablet_manager.cc:1929] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:04.023536 1179 log.cc:1199] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e9af06b4d79441968bda6bd7d67d76bf
I20251029 08:52:04.024106 1179 ts_tablet_manager.cc:1950] T e9af06b4d79441968bda6bd7d67d76bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.024137 1175 tablet_replica.cc:333] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.024362 1175 raft_consensus.cc:2243] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.024577 1175 raft_consensus.cc:2272] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.024871 1178 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.025393 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 56) took 1134 ms. Trace:
I20251029 08:52:04.025537 1179 rpcz_store.cc:276] 1029 08:52:02.890425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.891587 (+ 1162us) service_pool.cc:225] Handling call
1029 08:52:04.013103 (+1121516us) ts_tablet_manager.cc:1151] Deleting tablet e9af06b4d79441968bda6bd7d67d76bf
1029 08:52:04.013113 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.019489 (+ 6376us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.023207 (+ 3718us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.025277 (+ 2070us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27,"tablet-delete.queue_time_us":1102276}
I20251029 08:52:04.026125 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e9af06b4d79441968bda6bd7d67d76bf (table default.Table_0567a7bfaea44a6d8bd8807e6206939c [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:04.026520 1179 tablet_replica.cc:333] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.027168 1179 raft_consensus.cc:2243] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.027426 1179 raft_consensus.cc:2272] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.027650 1178 ts_tablet_manager.cc:1403] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.004s
I20251029 08:52:04.027642 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_806b95a12287444cbf2898b4e4c93f65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.028409 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_806b95a12287444cbf2898b4e4c93f65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.028596 1178 raft_consensus.cc:359] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.028797 1178 raft_consensus.cc:385] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.028800 1175 ts_tablet_manager.cc:1916] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.028869 1178 raft_consensus.cc:740] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.029141 1178 consensus_queue.cc:260] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.031301 1179 ts_tablet_manager.cc:1916] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.031309 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_53de717e75854922a7088385152a36af" } new_table_name: "default.Table_c0e519f0c5604145a2472a038ef78e46" modify_external_catalogs: true
I20251029 08:52:04.032002 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e0316d58fa5c49d0a294cd9a37632924"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.032575 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e18fb157b5f64bbb951cc5ca8c5e3936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.033177 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e18fb157b5f64bbb951cc5ca8c5e3936 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.033282 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ff2ff839cd5b4f62a2b81052c1c709b8" } new_table_name: "default.Table_ab2c1cce4615467595cb78eb6ca1b4ec" modify_external_catalogs: true
I20251029 08:52:04.035991 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.037001 1176 tablet_replica.cc:333] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.037201 1176 raft_consensus.cc:2243] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.037369 1176 raft_consensus.cc:2272] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.032577 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0316d58fa5c49d0a294cd9a37632924 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.039638 1176 ts_tablet_manager.cc:1916] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.040623 1178 ts_tablet_manager.cc:1434] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251029 08:52:04.048501 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 1557722a7c8041c88ee5321c0944e234
I20251029 08:52:04.049813 1176 ts_tablet_manager.cc:1929] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.050583 1043 tablet_service.cc:1505] Processing CreateTablet for tablet a20b64ff177241309e5d59e84f0bafa2 (DEFAULT_TABLE table=default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.050900 1183 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.051071 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20b64ff177241309e5d59e84f0bafa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.051069 1183 ts_tablet_manager.cc:1403] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.010s
I20251029 08:52:04.051209 1051 tablet_service.cc:1505] Processing CreateTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef (DEFAULT_TABLE table=default.Table_6fe27de4b7664cd5907bfb6507779a9b [id=f9c1873a2560495fa17fc61a1c1906b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.051600 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11998f3ebf3402ab3f729d04fd5e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.052157 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 1557722a7c8041c88ee5321c0944e234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.052095 1183 raft_consensus.cc:359] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.052315 1183 raft_consensus.cc:385] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.052448 1183 raft_consensus.cc:740] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.052706 1183 consensus_queue.cc:260] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.053351 1183 ts_tablet_manager.cc:1434] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:04.053715 1183 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.055589 1183 tablet_bootstrap.cc:654] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.057024 900 tablet_service.cc:1505] Processing CreateTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef (DEFAULT_TABLE table=default.Table_6fe27de4b7664cd5907bfb6507779a9b [id=f9c1873a2560495fa17fc61a1c1906b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.057230 1176 log.cc:1199] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:04.057487 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11998f3ebf3402ab3f729d04fd5e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.057632 1176 ts_tablet_manager.cc:1950] T 0d8a59f8f764466686a4b25c3b28596f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.058933 593 master_service.cc:954] ReplaceTablet: received request to replace tablet a232a4a898f04f339084c807990a14e1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.059484 1177 tablet_bootstrap.cc:492] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.059672 1177 ts_tablet_manager.cc:1403] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.002s
I20251029 08:52:04.060246 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 1557722a7c8041c88ee5321c0944e234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.060503 760 tablet_service.cc:1505] Processing CreateTablet for tablet a20b64ff177241309e5d59e84f0bafa2 (DEFAULT_TABLE table=default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.057024 902 tablet_service.cc:1505] Processing CreateTablet for tablet a20b64ff177241309e5d59e84f0bafa2 (DEFAULT_TABLE table=default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.060526 1177 raft_consensus.cc:359] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.060783 1177 raft_consensus.cc:385] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.060882 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20b64ff177241309e5d59e84f0bafa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.060915 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20b64ff177241309e5d59e84f0bafa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.060889 1177 raft_consensus.cc:740] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.061215 1177 consensus_queue.cc:260] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.061816 1177 ts_tablet_manager.cc:1434] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 08:52:04.062563 1177 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.064386 749 tablet_service.cc:1505] Processing CreateTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef (DEFAULT_TABLE table=default.Table_6fe27de4b7664cd5907bfb6507779a9b [id=f9c1873a2560495fa17fc61a1c1906b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.064802 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11998f3ebf3402ab3f729d04fd5e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.065609 1177 tablet_bootstrap.cc:654] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.068181 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 1557722a7c8041c88ee5321c0944e234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.069759 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0d8a59f8f764466686a4b25c3b28596f (table default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]) successfully deleted
I20251029 08:52:04.070639 1177 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.070830 1177 ts_tablet_manager.cc:1403] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20251029 08:52:04.071215 758 tablet_service.cc:1505] Processing CreateTablet for tablet 57e2b19a227740a28ae601d77e62597b (DEFAULT_TABLE table=default.Table_ff2ff839cd5b4f62a2b81052c1c709b8 [id=d283eaf060754468a34dfd1849f618ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.071630 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e2b19a227740a28ae601d77e62597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.073037 1177 raft_consensus.cc:359] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.073344 1177 raft_consensus.cc:385] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.073483 1177 raft_consensus.cc:740] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.073937 1177 consensus_queue.cc:260] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.074163 1175 ts_tablet_manager.cc:1929] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.075107 1175 log.cc:1199] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4b8068c1c7ce49fc85b660f40a7d1c58
I20251029 08:52:04.075697 1175 ts_tablet_manager.cc:1950] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.075766 1177 ts_tablet_manager.cc:1434] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 08:52:04.076515 1177 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.078305 1177 tablet_bootstrap.cc:654] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.079804 900 tablet_service.cc:1505] Processing CreateTablet for tablet 4cf906c73dc84403bcc3dece74a822bd (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.080307 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf906c73dc84403bcc3dece74a822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.080821 898 tablet_service.cc:1505] Processing CreateTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 (DEFAULT_TABLE table=default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.081148 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fdbe6681894ffa8264cdfa486f16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.084374 896 tablet_service.cc:1505] Processing CreateTablet for tablet a2947083e3e447d5addae5effd186eb8 (DEFAULT_TABLE table=default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.084757 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2947083e3e447d5addae5effd186eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.092576 762 tablet_service.cc:1505] Processing CreateTablet for tablet 4cf906c73dc84403bcc3dece74a822bd (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.093075 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf906c73dc84403bcc3dece74a822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.099330 747 tablet_service.cc:1505] Processing CreateTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 (DEFAULT_TABLE table=default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.100361 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fdbe6681894ffa8264cdfa486f16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.101649 756 tablet_service.cc:1505] Processing CreateTablet for tablet a2947083e3e447d5addae5effd186eb8 (DEFAULT_TABLE table=default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.102051 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2947083e3e447d5addae5effd186eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.102767 1177 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.103000 1177 ts_tablet_manager.cc:1403] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20251029 08:52:04.103892 1177 raft_consensus.cc:359] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.105571 1175 tablet_replica.cc:333] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.106041 1175 raft_consensus.cc:2243] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.106379 1175 raft_consensus.cc:2272] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.106206 1177 raft_consensus.cc:385] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.106667 1177 raft_consensus.cc:740] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.107028 1177 consensus_queue.cc:260] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.107237 1175 ts_tablet_manager.cc:1916] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.107409 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
I20251029 08:52:04.107599 1177 ts_tablet_manager.cc:1434] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:04.108386 1177 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.110798 1177 tablet_bootstrap.cc:654] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.114866 1175 ts_tablet_manager.cc:1929] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.115779 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 57e2b19a227740a28ae601d77e62597b (DEFAULT_TABLE table=default.Table_ff2ff839cd5b4f62a2b81052c1c709b8 [id=d283eaf060754468a34dfd1849f618ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.116506 1041 tablet_service.cc:1505] Processing CreateTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 (DEFAULT_TABLE table=default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.117655 1175 log.cc:1199] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9487a5ebf7ac404695ef85848278ec8d
I20251029 08:52:04.118427 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e2b19a227740a28ae601d77e62597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.118465 1183 tablet_bootstrap.cc:492] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.118611 1183 ts_tablet_manager.cc:1403] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.029s
I20251029 08:52:04.119424 1183 raft_consensus.cc:359] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.120157 1183 raft_consensus.cc:385] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.120182 1177 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.120245 1183 raft_consensus.cc:740] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.120348 1177 ts_tablet_manager.cc:1403] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20251029 08:52:04.120487 1183 consensus_queue.cc:260] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.121083 1177 raft_consensus.cc:359] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.121685 1183 ts_tablet_manager.cc:1434] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:04.123240 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 4cf906c73dc84403bcc3dece74a822bd (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.123670 1177 raft_consensus.cc:385] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.121318 1175 ts_tablet_manager.cc:1950] T 9487a5ebf7ac404695ef85848278ec8d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.124958 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9487a5ebf7ac404695ef85848278ec8d (table default.Table_a72fbddadb854e8aa8f5c07a56e0ae0c [id=2d44e62ba8374cee8a61818d460f6eab]) successfully deleted
I20251029 08:52:04.125339 1175 tablet_replica.cc:333] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.125556 1175 raft_consensus.cc:2243] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:04.125576 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 24ed199b700f4f92858c4edb95217d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ed199b700f4f92858c4edb95217d03 already in progress: creating tablet
I20251029 08:52:04.125769 1175 raft_consensus.cc:2272] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.125802 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 25b2e13184de4c25a5f14bc78d871bda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25b2e13184de4c25a5f14bc78d871bda already in progress: creating tablet
W20251029 08:52:04.126080 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 904523f7164849139cddb1100e95b824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904523f7164849139cddb1100e95b824 already in progress: creating tablet
I20251029 08:52:04.126529 1175 ts_tablet_manager.cc:1916] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.126930 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf906c73dc84403bcc3dece74a822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.131251 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fdbe6681894ffa8264cdfa486f16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.132090 1177 raft_consensus.cc:740] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.132659 1177 consensus_queue.cc:260] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.133678 1183 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.134776 1177 ts_tablet_manager.cc:1434] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251029 08:52:04.135015 902 tablet_service.cc:1505] Processing CreateTablet for tablet 57e2b19a227740a28ae601d77e62597b (DEFAULT_TABLE table=default.Table_ff2ff839cd5b4f62a2b81052c1c709b8 [id=d283eaf060754468a34dfd1849f618ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.135461 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e2b19a227740a28ae601d77e62597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.135891 1183 tablet_bootstrap.cc:654] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.135919 1177 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.137573 1177 tablet_bootstrap.cc:654] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.139616 1179 ts_tablet_manager.cc:1929] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.139727 1179 log.cc:1199] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a709caf6b7244a2d8ab45f6f578cf7cf
I20251029 08:52:04.140067 1179 ts_tablet_manager.cc:1950] T a709caf6b7244a2d8ab45f6f578cf7cf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.140862 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 57) took 1250 ms. Trace:
I20251029 08:52:04.141301 1179 rpcz_store.cc:276] 1029 08:52:02.890548 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.891612 (+ 1064us) service_pool.cc:225] Handling call
1029 08:52:04.026439 (+1134827us) ts_tablet_manager.cc:1151] Deleting tablet a709caf6b7244a2d8ab45f6f578cf7cf
1029 08:52:04.026448 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.054920 (+ 28472us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.139515 (+ 84595us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.140829 (+ 1314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113978}
I20251029 08:52:04.142946 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a709caf6b7244a2d8ab45f6f578cf7cf (table default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]) successfully deleted
I20251029 08:52:04.143347 900 tablet_service.cc:1505] Processing CreateTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f (DEFAULT_TABLE table=default.Table_d1d79c50455c4a05a0a804d863b09557 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.143373 749 tablet_service.cc:1505] Processing CreateTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f (DEFAULT_TABLE table=default.Table_d1d79c50455c4a05a0a804d863b09557 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.143435 1179 tablet_replica.cc:333] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.143754 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.143787 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.142532 1043 tablet_service.cc:1505] Processing CreateTablet for tablet a2947083e3e447d5addae5effd186eb8 (DEFAULT_TABLE table=default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.144266 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2947083e3e447d5addae5effd186eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.145948 1178 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.146307 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f (DEFAULT_TABLE table=default.Table_d1d79c50455c4a05a0a804d863b09557 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.146746 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.147571 1178 tablet_bootstrap.cc:654] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.149823 1176 tablet_replica.cc:333] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.150013 1176 raft_consensus.cc:2243] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.150216 1176 raft_consensus.cc:2272] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.151710 762 tablet_service.cc:1505] Processing CreateTablet for tablet 09759a50a82f4b758449da9fbbc1f72d (DEFAULT_TABLE table=default.Table_e4335ad852264f3382fd20c90284b5c1 [id=dc9c30acd58449e4bcb9217b628b59dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.152161 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09759a50a82f4b758449da9fbbc1f72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.144428 1179 raft_consensus.cc:2243] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.152532 1179 raft_consensus.cc:2272] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.154083 1175 ts_tablet_manager.cc:1929] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.154215 1175 log.cc:1199] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/07a7ee229f4d4b2fa6dd1f75b4aaff74
I20251029 08:52:04.154690 1175 ts_tablet_manager.cc:1950] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.155217 1179 ts_tablet_manager.cc:1916] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.155380 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74 (table default.Table_d881df3a5f1043079e80ac59d710e3ac [id=fe2e71ba07f34b5c9b68619573db2440]) successfully deleted
I20251029 08:52:04.155938 900 tablet_service.cc:1505] Processing CreateTablet for tablet 09759a50a82f4b758449da9fbbc1f72d (DEFAULT_TABLE table=default.Table_e4335ad852264f3382fd20c90284b5c1 [id=dc9c30acd58449e4bcb9217b628b59dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.156446 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09759a50a82f4b758449da9fbbc1f72d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.158610 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b4ceeaba655542dab2556d3988b6e841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ceeaba655542dab2556d3988b6e841 already in progress: creating tablet
W20251029 08:52:04.159765 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cf4934c3842a463e857262ceed37e0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf4934c3842a463e857262ceed37e0bc already in progress: creating tablet
I20251029 08:52:04.160529 1175 tablet_replica.cc:333] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.161168 1175 raft_consensus.cc:2243] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.162011 1175 raft_consensus.cc:2272] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.167812 1176 ts_tablet_manager.cc:1916] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.170362 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 09759a50a82f4b758449da9fbbc1f72d (DEFAULT_TABLE table=default.Table_e4335ad852264f3382fd20c90284b5c1 [id=dc9c30acd58449e4bcb9217b628b59dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.170866 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09759a50a82f4b758449da9fbbc1f72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.172586 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:04.175374 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_985f7f5558144c058beb6d77045f1924" } new_table_name: "default.Table_1c1e2303aee3434c959bd347637bae57" modify_external_catalogs: true
I20251029 08:52:04.177194 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e4335ad852264f3382fd20c90284b5c1" } new_table_name: "default.Table_0f93256544fb4b56adbbcaf8c50ecb42" modify_external_catalogs: true
I20251029 08:52:04.178416 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_971780f9a8d5435a9f6a94d5727f5119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.178939 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_971780f9a8d5435a9f6a94d5727f5119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.175469 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6899fb76bad448cd9d1ca4299c5f7562"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.180310 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6899fb76bad448cd9d1ca4299c5f7562 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.181874 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:04.183449 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57e2b19a227740a28ae601d77e62597b consensus state has no leader
I20251029 08:52:04.185107 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:04.187301 1175 ts_tablet_manager.cc:1916] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.190950 1179 ts_tablet_manager.cc:1929] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.191558 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a709caf6b7244a2d8ab45f6f578cf7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.191985 1178 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.192442 1178 ts_tablet_manager.cc:1403] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20251029 08:52:04.193784 1178 raft_consensus.cc:359] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:04.195631 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09759a50a82f4b758449da9fbbc1f72d consensus state has no leader
W20251029 08:52:04.195865 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4e1b6b6359b437aa295b473fa42dce1 consensus state has no leader
I20251029 08:52:04.195878 1178 raft_consensus.cc:385] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.195981 1178 raft_consensus.cc:740] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.196256 1178 consensus_queue.cc:260] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.198416 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 06eb710809754e7abd00a345e8ae591a
I20251029 08:52:04.199784 1178 ts_tablet_manager.cc:1434] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:04.199754 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9df4bbf8684046b9876226a4a342654c" } new_table_name: "default.Table_cd11e136b8054bdfb9c12527b0fc5777" modify_external_catalogs: true
I20251029 08:52:04.200290 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_80b0f80eda77452b959592ce98b44953" } modify_external_catalogs: true
I20251029 08:52:04.200474 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_80b0f80eda77452b959592ce98b44953" } modify_external_catalogs: true
I20251029 08:52:04.202478 1178 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.202878 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_14e650112f054d8ea69743954a31706a" } modify_external_catalogs: true
I20251029 08:52:04.203028 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_14e650112f054d8ea69743954a31706a" } modify_external_catalogs: true
I20251029 08:52:04.204269 1178 tablet_bootstrap.cc:654] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.204448 1179 log.cc:1199] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4b8068c1c7ce49fc85b660f40a7d1c58
I20251029 08:52:04.205029 1179 ts_tablet_manager.cc:1950] T 4b8068c1c7ce49fc85b660f40a7d1c58 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.207124 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 52) took 1317 ms. Trace:
I20251029 08:52:04.208081 1179 rpcz_store.cc:276] 1029 08:52:02.890003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:02.891221 (+ 1218us) service_pool.cc:225] Handling call
1029 08:52:04.143346 (+1252125us) ts_tablet_manager.cc:1151] Deleting tablet 4b8068c1c7ce49fc85b660f40a7d1c58
1029 08:52:04.143360 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.170649 (+ 27289us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.190833 (+ 20184us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.207068 (+ 16235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228770}
I20251029 08:52:04.208643 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4b8068c1c7ce49fc85b660f40a7d1c58 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
I20251029 08:52:04.209190 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 67) took 1131 ms. Trace:
I20251029 08:52:04.210527 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b7aebe05237b4357bab8470b1e161bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.211150 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7aebe05237b4357bab8470b1e161bc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.212411 1179 rpcz_store.cc:276] 1029 08:52:03.077407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.079356 (+ 1949us) service_pool.cc:225] Handling call
1029 08:52:04.209013 (+1129657us) ts_tablet_manager.cc:1151] Deleting tablet 6b8a4b96d3c748daa0f467768b2b7abd
1029 08:52:04.209026 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.209135 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129425}
I20251029 08:52:04.213055 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 68) took 1135 ms. Trace:
W20251029 08:52:04.213069 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6b8a4b96d3c748daa0f467768b2b7abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b8a4b96d3c748daa0f467768b2b7abd already in progress: creating tablet
I20251029 08:52:04.213090 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 954c264e600a46f19db40dde5180787d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.207608 760 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.213270 1179 rpcz_store.cc:276] 1029 08:52:03.077538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.079611 (+ 2073us) service_pool.cc:225] Handling call
1029 08:52:04.212884 (+1133273us) ts_tablet_manager.cc:1151] Deleting tablet 8ded7b3943e544e585fe52433ff7555f
1029 08:52:04.212894 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.213002 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133106}
I20251029 08:52:04.207772 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:04.213878 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8ded7b3943e544e585fe52433ff7555f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ded7b3943e544e585fe52433ff7555f already in progress: creating tablet
I20251029 08:52:04.214255 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 79) took 1057 ms. Trace:
I20251029 08:52:04.214393 1179 rpcz_store.cc:276] 1029 08:52:03.157135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.157278 (+ 143us) service_pool.cc:225] Handling call
1029 08:52:04.214107 (+1056829us) ts_tablet_manager.cc:1151] Deleting tablet e72f64830e924fa48e2fa4cf81ccbba3
1029 08:52:04.214119 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.214206 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056561}
I20251029 08:52:04.214495 898 tablet_service.cc:1552] Processing DeleteTablet for tablet a232a4a898f04f339084c807990a14e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 61b07eb4befa41f3a1b5f2e9ebc2277c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.214644 1177 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.214854 1177 ts_tablet_manager.cc:1403] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.004s
I20251029 08:52:04.214895 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a232a4a898f04f339084c807990a14e1
I20251029 08:52:04.215003 1179 tablet_replica.cc:333] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.215248 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.215396 593 catalog_manager.cc:6719] ReplaceTablet: tablet a232a4a898f04f339084c807990a14e1 deleted and replaced by tablet 61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:04.215600 1179 raft_consensus.cc:2243] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.215656 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 3c9a18c9e63d4b5e9a42fe053b187b08 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.216184 1179 raft_consensus.cc:2272] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.215981 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e72f64830e924fa48e2fa4cf81ccbba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e72f64830e924fa48e2fa4cf81ccbba3 already in progress: creating tablet
I20251029 08:52:04.216231 1177 raft_consensus.cc:359] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.216775 756 tablet_service.cc:1552] Processing DeleteTablet for tablet a232a4a898f04f339084c807990a14e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 61b07eb4befa41f3a1b5f2e9ebc2277c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.216841 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet a232a4a898f04f339084c807990a14e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 61b07eb4befa41f3a1b5f2e9ebc2277c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.216482 899 tablet_service.cc:1552] Processing DeleteTablet for tablet a232a4a898f04f339084c807990a14e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 61b07eb4befa41f3a1b5f2e9ebc2277c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.217620 1177 raft_consensus.cc:385] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.217744 1177 raft_consensus.cc:740] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.218047 1177 consensus_queue.cc:260] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.218333 1179 ts_tablet_manager.cc:1916] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.217386 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:04.218974 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0b459b366cb402db2e7e8fee2b3e855 consensus state has no leader
I20251029 08:52:04.219576 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f65f8ca443e48a48d7ef4a474dc95b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.217458 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:04.220814 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8cb0c6c6a1c4877a89476d4822f48db consensus state has no leader
I20251029 08:52:04.220849 1177 ts_tablet_manager.cc:1434] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:04.221246 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_71c5f2a1b15b4d1e95c616783cd6b837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.221624 1177 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:04.221841 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71c5f2a1b15b4d1e95c616783cd6b837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.225428 1177 tablet_bootstrap.cc:654] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.225659 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.225970 1175 ts_tablet_manager.cc:1929] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.226447 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4b89cba8e4a642dabf87ae4f9a509895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.226984 902 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:04.227128 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b89cba8e4a642dabf87ae4f9a509895 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.227665 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 8ded7b3943e544e585fe52433ff7555f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.229717 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_affbc133198140a2b050f47a2d51c45d" } modify_external_catalogs: true
I20251029 08:52:04.229889 1175 log.cc:1199] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/1557722a7c8041c88ee5321c0944e234
I20251029 08:52:04.229887 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_affbc133198140a2b050f47a2d51c45d" } modify_external_catalogs: true
I20251029 08:52:04.230370 1175 ts_tablet_manager.cc:1950] T 1557722a7c8041c88ee5321c0944e234 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.231385 1178 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.231642 1178 ts_tablet_manager.cc:1403] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251029 08:52:04.232626 1178 raft_consensus.cc:359] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.232888 1178 raft_consensus.cc:385] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.233028 1178 raft_consensus.cc:740] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.233355 1178 consensus_queue.cc:260] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.234128 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_742704efe2d14569b8a72c8e7d09c736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.234933 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_742704efe2d14569b8a72c8e7d09c736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.235471 1178 ts_tablet_manager.cc:1434] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:04.232982 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_796d0f3f111e416aa8c95689e9d80a73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.237138 1178 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:04.237337 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_796d0f3f111e416aa8c95689e9d80a73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.237344 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:04.238149 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 1557722a7c8041c88ee5321c0944e234 (table default.Table_3a8b8f91860e419e8538f028e57603fd [id=224c13af437a46fab3f2bd9b3d20f523]) successfully deleted
I20251029 08:52:04.239365 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.239697 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.240548 1177 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.240792 1177 ts_tablet_manager.cc:1403] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20251029 08:52:04.241730 1183 tablet_bootstrap.cc:492] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.241765 1177 raft_consensus.cc:359] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.241951 1183 ts_tablet_manager.cc:1403] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.010s
I20251029 08:52:04.242017 1177 raft_consensus.cc:385] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.242184 1177 raft_consensus.cc:740] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.242635 1177 consensus_queue.cc:260] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.243311 1177 ts_tablet_manager.cc:1434] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:04.243408 1175 tablet_replica.cc:333] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.243610 1175 raft_consensus.cc:2243] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.244197 597 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 67c42b3f0dce41bda9c981aa8b1acbeb
I20251029 08:52:04.244722 1183 raft_consensus.cc:359] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.244933 1183 raft_consensus.cc:385] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.245016 1183 raft_consensus.cc:740] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.245265 762 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.245265 1183 consensus_queue.cc:260] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.247570 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ca96f0b4ad79470a8599b4e0e8cac23d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.248107 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca96f0b4ad79470a8599b4e0e8cac23d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.248570 1176 ts_tablet_manager.cc:1929] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.248716 1176 log.cc:1199] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/1557722a7c8041c88ee5321c0944e234
I20251029 08:52:04.243804 1175 raft_consensus.cc:2272] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.249130 1176 ts_tablet_manager.cc:1950] T 1557722a7c8041c88ee5321c0944e234 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.249665 1183 ts_tablet_manager.cc:1434] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 08:52:04.249855 1175 ts_tablet_manager.cc:1916] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.250157 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 1557722a7c8041c88ee5321c0944e234 (table default.Table_3a8b8f91860e419e8538f028e57603fd [id=224c13af437a46fab3f2bd9b3d20f523]) successfully deleted
I20251029 08:52:04.253307 1183 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.253829 1178 tablet_bootstrap.cc:654] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.254384 1176 tablet_replica.cc:333] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:04.254588 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0d8a59f8f764466686a4b25c3b28596f because the tablet was not found. No further retry: Not found: Tablet not found: 0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:04.260100 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d8a59f8f764466686a4b25c3b28596f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.263291 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165" } modify_external_catalogs: true
I20251029 08:52:04.263509 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165" } modify_external_catalogs: true
I20251029 08:52:04.263530 1179 ts_tablet_manager.cc:1929] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.263665 1179 log.cc:1199] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/de8f868095a642f6a5d70c30bb74105c
I20251029 08:52:04.264082 1179 ts_tablet_manager.cc:1950] T de8f868095a642f6a5d70c30bb74105c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.264226 1177 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.264974 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 80) took 1107 ms. Trace:
I20251029 08:52:04.265130 1179 rpcz_store.cc:276] 1029 08:52:03.157302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.157579 (+ 277us) service_pool.cc:225] Handling call
1029 08:52:04.214925 (+1057346us) ts_tablet_manager.cc:1151] Deleting tablet de8f868095a642f6a5d70c30bb74105c
1029 08:52:04.214934 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.241279 (+ 26345us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.263365 (+ 22086us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.264934 (+ 1569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057048}
I20251029 08:52:04.265638 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet de8f868095a642f6a5d70c30bb74105c (table default.Table_31cdca53404a4a36a5ee867f8aa53bfc [id=520b9919efc542c095d65a47ce1823ec]) successfully deleted
I20251029 08:52:04.266902 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:40435: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111),
I20251029 08:52:04.267059 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 08:52:04.267704 1183 tablet_bootstrap.cc:654] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.268591 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 954c264e600a46f19db40dde5180787d
I20251029 08:52:04.269065 588 catalog_manager.cc:6719] ReplaceTablet: tablet 954c264e600a46f19db40dde5180787d deleted and replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:04.270073 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a2947083e3e447d5addae5effd186eb8
I20251029 08:52:04.271893 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.273033 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.273639 1176 raft_consensus.cc:2243] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.273872 1176 raft_consensus.cc:2272] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.274466 1193 raft_consensus.cc:493] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.274654 1193 raft_consensus.cc:515] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.275336 1193 leader_election.cc:290] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.276115 1176 ts_tablet_manager.cc:1916] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.276664 1177 tablet_bootstrap.cc:654] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.276703 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ded7b3943e544e585fe52433ff7555f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:04.277572 861 leader_election.cc:343] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:04.277786 861 leader_election.cc:343] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 8ded7b3943e544e585fe52433ff7555f
I20251029 08:52:04.277920 861 leader_election.cc:304] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:04.278786 1177 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.278952 1177 ts_tablet_manager.cc:1403] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20251029 08:52:04.278995 1178 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.279134 1178 ts_tablet_manager.cc:1403] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.003s
I20251029 08:52:04.279186 1193 raft_consensus.cc:2749] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:04.279780 1177 raft_consensus.cc:359] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.279896 1178 raft_consensus.cc:359] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.280131 1178 raft_consensus.cc:385] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.280254 1178 raft_consensus.cc:740] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.280548 1178 consensus_queue.cc:260] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.280754 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.280973 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.281350 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.281656 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.282940 1178 ts_tablet_manager.cc:1434] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:04.279989 1177 raft_consensus.cc:385] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.283826 1177 raft_consensus.cc:740] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.284265 1178 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.284349 1177 consensus_queue.cc:260] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.284824 1177 ts_tablet_manager.cc:1434] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:04.285317 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 82) took 1108 ms. Trace:
I20251029 08:52:04.285955 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.286105 1179 rpcz_store.cc:276] 1029 08:52:03.186673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.186729 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:04.285039 (+1098310us) ts_tablet_manager.cc:1151] Deleting tablet e899e183e255498b83a6d740af98de42
1029 08:52:04.285054 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.285263 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087456}
W20251029 08:52:04.286592 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.286881 1177 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.287189 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 81) took 1117 ms. Trace:
I20251029 08:52:04.288605 1179 rpcz_store.cc:276] 1029 08:52:03.169837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.169914 (+ 77us) service_pool.cc:225] Handling call
1029 08:52:04.287052 (+1117138us) ts_tablet_manager.cc:1151] Deleting tablet cc1d1d5b2a9e4051bbcd25be1578ee73
1029 08:52:04.287062 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.287157 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084903}
I20251029 08:52:04.289449 1175 ts_tablet_manager.cc:1929] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.290208 1179 tablet_replica.cc:333] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.290655 1175 log.cc:1199] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a709caf6b7244a2d8ab45f6f578cf7cf
I20251029 08:52:04.290843 1178 tablet_bootstrap.cc:654] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:04.291489 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1d1d5b2a9e4051bbcd25be1578ee73 already in progress: creating tablet
W20251029 08:52:04.291734 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet db9cf2a8f2254a38a725bcd3c83e8509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9cf2a8f2254a38a725bcd3c83e8509 already in progress: creating tablet
I20251029 08:52:04.292397 1179 raft_consensus.cc:2243] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.292624 1179 raft_consensus.cc:2272] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.293545 1179 ts_tablet_manager.cc:1916] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.297096 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:04.298079 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.300261 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.283001 1176 ts_tablet_manager.cc:1929] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.301813 1176 log.cc:1199] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e899e183e255498b83a6d740af98de42
I20251029 08:52:04.301942 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.302683 1176 ts_tablet_manager.cc:1950] T e899e183e255498b83a6d740af98de42 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.303787 1177 tablet_bootstrap.cc:654] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.303982 1175 ts_tablet_manager.cc:1950] T a709caf6b7244a2d8ab45f6f578cf7cf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.305299 1175 tablet_replica.cc:333] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.305469 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a709caf6b7244a2d8ab45f6f578cf7cf (table default.Table_6ede2eb7ed6342c5a0b3e72c34c72cbd [id=0ae87561c6c14a03bb01ce74b6774e99]) successfully deleted
I20251029 08:52:04.305644 1175 raft_consensus.cc:2243] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.305840 1175 raft_consensus.cc:2272] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.306739 1175 ts_tablet_manager.cc:1916] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.303745 1178 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.307999 1183 tablet_bootstrap.cc:492] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.308205 1183 ts_tablet_manager.cc:1403] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.005s
I20251029 08:52:04.308286 1178 ts_tablet_manager.cc:1403] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20251029 08:52:04.309083 1183 raft_consensus.cc:359] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.309372 1183 raft_consensus.cc:385] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.309512 1183 raft_consensus.cc:740] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.309798 1183 consensus_queue.cc:260] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.312997 1183 ts_tablet_manager.cc:1434] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 08:52:04.312887 1178 raft_consensus.cc:359] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.313176 1178 raft_consensus.cc:385] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.313295 1178 raft_consensus.cc:740] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.313728 1178 consensus_queue.cc:260] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.315016 1179 ts_tablet_manager.cc:1929] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.316146 1179 log.cc:1199] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4f65f8ca443e48a48d7ef4a474dc95b9
I20251029 08:52:04.317332 1175 ts_tablet_manager.cc:1929] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.317379 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e899e183e255498b83a6d740af98de42 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
I20251029 08:52:04.317867 1179 ts_tablet_manager.cc:1950] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.318543 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 13eeb2e8951b421e9d6f8f228a956e3d (table default.Table_0567a7bfaea44a6d8bd8807e6206939c [id=407839d19657473883ad497873b78fb2]). Expected version 3 got 0
I20251029 08:52:04.319259 1179 tablet_replica.cc:333] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.319458 1179 raft_consensus.cc:2243] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.319890 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (table default.Table_7966c1de42494873aa075b0d9185824d [id=9b51cfa4d70542a280cc72c9528c45d4]) successfully deleted
I20251029 08:52:04.319937 1175 log.cc:1199] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a232a4a898f04f339084c807990a14e1
W20251029 08:52:04.320187 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet be476c9994ed4dc28f606955b16bc4de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be476c9994ed4dc28f606955b16bc4de already in progress: creating tablet
I20251029 08:52:04.320358 1175 ts_tablet_manager.cc:1950] T a232a4a898f04f339084c807990a14e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:04.321796 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 06eb710809754e7abd00a345e8ae591a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06eb710809754e7abd00a345e8ae591a already in progress: creating tablet
I20251029 08:52:04.321924 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a232a4a898f04f339084c807990a14e1 (table default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]) successfully deleted
I20251029 08:52:04.322265 1179 raft_consensus.cc:2272] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.321172 1176 tablet_replica.cc:333] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.322537 1176 raft_consensus.cc:2243] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.322640 1177 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.322741 1176 raft_consensus.cc:2272] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.322777 1177 ts_tablet_manager.cc:1403] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20251029 08:52:04.315095 1183 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.323050 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 13eeb2e8951b421e9d6f8f228a956e3d (table default.Table_0567a7bfaea44a6d8bd8807e6206939c [id=407839d19657473883ad497873b78fb2]). Expected version 3 got 0
I20251029 08:52:04.323964 1177 raft_consensus.cc:359] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.324234 1177 raft_consensus.cc:385] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.324442 1177 raft_consensus.cc:740] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.324581 749 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.324748 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:04.325182 1178 ts_tablet_manager.cc:1434] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251029 08:52:04.325527 1177 consensus_queue.cc:260] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.326447 1183 tablet_bootstrap.cc:654] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.328266 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.329358 1176 ts_tablet_manager.cc:1916] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.330055 1179 ts_tablet_manager.cc:1916] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.330364 1175 tablet_replica.cc:333] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.330597 1175 raft_consensus.cc:2243] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.330857 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:04.331465 1175 raft_consensus.cc:2272] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.329553 1178 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.332221 1175 ts_tablet_manager.cc:1916] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.332530 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.333333 1177 ts_tablet_manager.cc:1434] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251029 08:52:04.333859 1178 tablet_bootstrap.cc:654] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.335247 1177 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.338979 749 tablet_service.cc:1505] Processing CreateTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (DEFAULT_TABLE table=default.Table_806b95a12287444cbf2898b4e4c93f65 [id=a2015498bb844e87af722a69bcbf616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.339020 1041 tablet_service.cc:1505] Processing CreateTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (DEFAULT_TABLE table=default.Table_806b95a12287444cbf2898b4e4c93f65 [id=a2015498bb844e87af722a69bcbf616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.339318 1177 tablet_bootstrap.cc:654] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.339423 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.339479 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.339757 900 tablet_service.cc:1505] Processing CreateTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (DEFAULT_TABLE table=default.Table_806b95a12287444cbf2898b4e4c93f65 [id=a2015498bb844e87af722a69bcbf616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.340140 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.348764 749 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (DEFAULT_TABLE table=default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b [id=4856bf7f477c47d58060d1728c544aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.349318 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.349357 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (DEFAULT_TABLE table=default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b [id=4856bf7f477c47d58060d1728c544aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.349790 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.352522 1183 tablet_bootstrap.cc:492] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.352806 1183 ts_tablet_manager.cc:1403] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251029 08:52:04.353909 1177 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.353801 1183 raft_consensus.cc:359] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.354002 1183 raft_consensus.cc:385] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.354171 1177 ts_tablet_manager.cc:1403] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20251029 08:52:04.355094 1177 raft_consensus.cc:359] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.355352 1177 raft_consensus.cc:385] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.355486 1177 raft_consensus.cc:740] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.355849 1177 consensus_queue.cc:260] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.356482 1177 ts_tablet_manager.cc:1434] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:04.357307 1177 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.357455 1183 raft_consensus.cc:740] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.357806 1183 consensus_queue.cc:260] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.358491 900 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (DEFAULT_TABLE table=default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b [id=4856bf7f477c47d58060d1728c544aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.358712 1183 ts_tablet_manager.cc:1434] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:04.358935 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.359264 1183 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.361876 1183 tablet_bootstrap.cc:654] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.362743 762 tablet_service.cc:1505] Processing CreateTablet for tablet 64f73597d1764540bcaf472d3e890ee8 (DEFAULT_TABLE table=default.Table_29ca0b751907426496780bf44111f5d2 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.363178 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f73597d1764540bcaf472d3e890ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.364784 1266 raft_consensus.cc:493] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.364944 1266 raft_consensus.cc:515] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.365651 1266 leader_election.cc:290] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.366557 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4deeec492a2545ca86583c6ea899a0ab (DEFAULT_TABLE table=default.Table_c4b692fd4fae4d93bd787b008adc3b20 [id=58d71d681f8b4cd4967daf7d53eb5191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.366968 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4deeec492a2545ca86583c6ea899a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.365770 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 64f73597d1764540bcaf472d3e890ee8 (DEFAULT_TABLE table=default.Table_29ca0b751907426496780bf44111f5d2 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.366757 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ded7b3943e544e585fe52433ff7555f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.367837 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4ceeaba655542dab2556d3988b6e841" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:04.371227 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f73597d1764540bcaf472d3e890ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.368950 1179 ts_tablet_manager.cc:1929] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.372529 1176 ts_tablet_manager.cc:1929] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.372581 1179 log.cc:1199] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/07a7ee229f4d4b2fa6dd1f75b4aaff74
I20251029 08:52:04.372664 1176 log.cc:1199] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a232a4a898f04f339084c807990a14e1
I20251029 08:52:04.373026 1176 ts_tablet_manager.cc:1950] T a232a4a898f04f339084c807990a14e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.373211 1179 ts_tablet_manager.cc:1950] T 07a7ee229f4d4b2fa6dd1f75b4aaff74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.373687 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.374030 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a232a4a898f04f339084c807990a14e1 (table default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]) successfully deleted
I20251029 08:52:04.374142 593 catalog_manager.cc:6719] ReplaceTablet: tablet 3c9a18c9e63d4b5e9a42fe053b187b08 deleted and replaced by tablet 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:04.374233 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 95) took 1001 ms. Trace:
I20251029 08:52:04.374347 1179 rpcz_store.cc:276] 1029 08:52:03.372659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:03.372779 (+ 120us) service_pool.cc:225] Handling call
1029 08:52:04.319168 (+946389us) ts_tablet_manager.cc:1151] Deleting tablet 07a7ee229f4d4b2fa6dd1f75b4aaff74
1029 08:52:04.319190 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:04.338556 (+ 19366us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.368825 (+ 30269us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:04.374181 (+ 5356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":938980}
W20251029 08:52:04.374969 720 leader_election.cc:343] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.374750 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbce5d595364b4bad36d59fb532c237) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.375561 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbce5d595364b4bad36d59fb532c237) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:04.375828 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a7fdbe6681894ffa8264cdfa486f16e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7fdbe6681894ffa8264cdfa486f16e6 already in progress: creating tablet
W20251029 08:52:04.376204 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 51bb01c123c44fb69b2f174fca5d7a13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51bb01c123c44fb69b2f174fca5d7a13 already in progress: creating tablet
W20251029 08:52:04.376415 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6e2c32b84201470aa0dd697a5da12117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2c32b84201470aa0dd697a5da12117 already in progress: creating tablet
W20251029 08:52:04.376699 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 06c08679f1e746538bfa2c80d4fbdba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c08679f1e746538bfa2c80d4fbdba3 already in progress: creating tablet
I20251029 08:52:04.376864 1176 tablet_replica.cc:333] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:04.376901 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4e808bff7558472099587a0f95c002ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e808bff7558472099587a0f95c002ff already in progress: creating tablet
W20251029 08:52:04.377148 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6426b5a4f5784b3a97475cb23199f916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6426b5a4f5784b3a97475cb23199f916 already in progress: creating tablet
W20251029 08:52:04.377539 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 954c264e600a46f19db40dde5180787d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 954c264e600a46f19db40dde5180787d already in progress: creating tablet
I20251029 08:52:04.377655 1176 raft_consensus.cc:2243] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:04.377792 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a2947083e3e447d5addae5effd186eb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2947083e3e447d5addae5effd186eb8 already in progress: creating tablet
I20251029 08:52:04.377851 1176 raft_consensus.cc:2272] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.378031 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3a28a4aa361468c9fa9e72ccd6e7b7f already in progress: creating tablet
I20251029 08:52:04.379118 1176 ts_tablet_manager.cc:1916] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.381385 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4ceeaba655542dab2556d3988b6e841" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:04.382225 718 leader_election.cc:343] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.382390 718 leader_election.cc:304] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:04.382653 718 raft_consensus.cc:2686] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:04.383744 902 tablet_service.cc:1505] Processing CreateTablet for tablet 64f73597d1764540bcaf472d3e890ee8 (DEFAULT_TABLE table=default.Table_29ca0b751907426496780bf44111f5d2 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.384065 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9a18c9e63d4b5e9a42fe053b187b08 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbce5d595364b4bad36d59fb532c237) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.384248 896 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.384433 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 4deeec492a2545ca86583c6ea899a0ab (DEFAULT_TABLE table=default.Table_c4b692fd4fae4d93bd787b008adc3b20 [id=58d71d681f8b4cd4967daf7d53eb5191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.384588 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f73597d1764540bcaf472d3e890ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.385972 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1e34f0b88fa4485094c008516dd01ab2" } new_table_name: "default.Table_73193142c4d04b1f83e12f1ac807a71f" modify_external_catalogs: true
I20251029 08:52:04.391211 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4deeec492a2545ca86583c6ea899a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.391330 1177 tablet_bootstrap.cc:654] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.393965 1175 ts_tablet_manager.cc:1929] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.394085 1175 log.cc:1199] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/8ded7b3943e544e585fe52433ff7555f
W20251029 08:52:04.394091 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb6b86f39c994ea5930a6a970830093f consensus state has no leader
I20251029 08:52:04.394357 1178 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.394449 1175 ts_tablet_manager.cc:1950] T 8ded7b3943e544e585fe52433ff7555f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.394511 1178 ts_tablet_manager.cc:1403] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.008s
I20251029 08:52:04.395462 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 8ded7b3943e544e585fe52433ff7555f (table default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3 [id=5124fdf717764856bff708bedd9656c8]) successfully deleted
W20251029 08:52:04.396134 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a232a4a898f04f339084c807990a14e1 because the tablet was not found. No further retry: Not found: Tablet not found: a232a4a898f04f339084c807990a14e1
I20251029 08:52:04.395345 1178 raft_consensus.cc:359] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.396337 1178 raft_consensus.cc:385] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.396407 1175 tablet_replica.cc:333] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.396437 1178 raft_consensus.cc:740] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.396610 1175 raft_consensus.cc:2243] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.396754 1175 raft_consensus.cc:2272] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.396710 1178 consensus_queue.cc:260] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.397485 1175 ts_tablet_manager.cc:1916] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.399488 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bb8cc7be97b8484f81dfddb6bb10e37a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.399888 1178 ts_tablet_manager.cc:1434] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251029 08:52:04.400137 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb8cc7be97b8484f81dfddb6bb10e37a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.401316 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_806b95a12287444cbf2898b4e4c93f65" } new_table_name: "default.Table_15261ce913ae430d8ea68e18a42df64a" modify_external_catalogs: true
I20251029 08:52:04.402100 1178 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.404003 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0e93e5470b774809972ce32757764331" } new_table_name: "default.Table_c630c3859f114c0d871851d8f87a6544" modify_external_catalogs: true
I20251029 08:52:04.404348 1178 tablet_bootstrap.cc:654] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.407963 756 tablet_service.cc:1552] Processing DeleteTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.410058 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4adb698e66ef4612946087c05b279db2" } modify_external_catalogs: true
I20251029 08:52:04.410324 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4adb698e66ef4612946087c05b279db2" } modify_external_catalogs: true
I20251029 08:52:04.411079 1175 ts_tablet_manager.cc:1929] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.411257 1175 log.cc:1199] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:04.411659 1175 ts_tablet_manager.cc:1950] T 0d8a59f8f764466686a4b25c3b28596f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.391556 1179 tablet_replica.cc:333] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.412343 1179 raft_consensus.cc:2243] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.412580 1179 raft_consensus.cc:2272] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.413317 1179 ts_tablet_manager.cc:1916] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.413552 1176 ts_tablet_manager.cc:1929] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.413712 1176 log.cc:1199] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:04.414062 1176 ts_tablet_manager.cc:1950] T b4ceeaba655542dab2556d3988b6e841 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.415062 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b4ceeaba655542dab2556d3988b6e841 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:04.415432 1178 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.415580 1178 ts_tablet_manager.cc:1403] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20251029 08:52:04.416438 1178 raft_consensus.cc:359] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.416574 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 0d8a59f8f764466686a4b25c3b28596f (table default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]) successfully deleted
I20251029 08:52:04.416954 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e735241cb8a04d689487db8e497b2d31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.417176 1178 raft_consensus.cc:385] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.417219 1175 tablet_replica.cc:333] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.417271 1178 raft_consensus.cc:740] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.417408 1175 raft_consensus.cc:2243] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:04.417573 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e735241cb8a04d689487db8e497b2d31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.417580 1175 raft_consensus.cc:2272] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.417536 1178 consensus_queue.cc:260] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.418915 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2eb98ef2bcb744d9b4560b816d57442d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.419241 1176 tablet_replica.cc:333] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.419415 1176 raft_consensus.cc:2243] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:04.419431 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eb98ef2bcb744d9b4560b816d57442d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.419587 1176 raft_consensus.cc:2272] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.421066 1176 ts_tablet_manager.cc:1916] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.424399 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.424907 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c4b692fd4fae4d93bd787b008adc3b20" } new_table_name: "default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e" modify_external_catalogs: true
I20251029 08:52:04.425043 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1d79c50455c4a05a0a804d863b09557" } new_table_name: "default.Table_3137cb7a7def4c5693f750cec7eb3f47" modify_external_catalogs: true
I20251029 08:52:04.421092 1178 ts_tablet_manager.cc:1434] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 08:52:04.426162 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 4cf906c73dc84403bcc3dece74a822bd from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.427188 1178 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.428339 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 453911c5a29f43e7a21fa2c47ef264e8 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.431123 1177 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.431365 1177 ts_tablet_manager.cc:1403] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.003s
I20251029 08:52:04.431502 1175 ts_tablet_manager.cc:1916] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.432399 1177 raft_consensus.cc:359] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.432657 1177 raft_consensus.cc:385] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.432796 1177 raft_consensus.cc:740] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.433115 1177 consensus_queue.cc:260] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.434511 1176 ts_tablet_manager.cc:1929] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.434640 1176 log.cc:1199] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/06c08679f1e746538bfa2c80d4fbdba3
I20251029 08:52:04.435010 1179 ts_tablet_manager.cc:1929] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.435164 1179 log.cc:1199] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/710e99b4116744289f9a2130f65f4b94
I20251029 08:52:04.435537 1179 ts_tablet_manager.cc:1950] T 710e99b4116744289f9a2130f65f4b94 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.437482 1177 ts_tablet_manager.cc:1434] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:04.439078 1177 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.440987 1178 tablet_bootstrap.cc:654] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.441720 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 710e99b4116744289f9a2130f65f4b94 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:04.441818 900 tablet_service.cc:1505] Processing CreateTablet for tablet 4deeec492a2545ca86583c6ea899a0ab (DEFAULT_TABLE table=default.Table_c4b692fd4fae4d93bd787b008adc3b20 [id=58d71d681f8b4cd4967daf7d53eb5191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.443017 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4deeec492a2545ca86583c6ea899a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.442814 756 tablet_service.cc:1505] Processing CreateTablet for tablet 7792ee223dbe43aa82990541caced045 (DEFAULT_TABLE table=default.Table_e18fb157b5f64bbb951cc5ca8c5e3936 [id=335946e15b7c424ba3b769c46bd76ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.443826 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7792ee223dbe43aa82990541caced045. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.443784 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 7792ee223dbe43aa82990541caced045 (DEFAULT_TABLE table=default.Table_e18fb157b5f64bbb951cc5ca8c5e3936 [id=335946e15b7c424ba3b769c46bd76ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.444780 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7792ee223dbe43aa82990541caced045. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.444938 762 tablet_service.cc:1505] Processing CreateTablet for tablet e687f593eead488db4bb445a63beeec7 (DEFAULT_TABLE table=default.Table_4b21589e83794d8e981ca8deeaf26b9f [id=a23a597c8fd343769b517507845104ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.445376 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e687f593eead488db4bb445a63beeec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.450476 1176 ts_tablet_manager.cc:1950] T 06c08679f1e746538bfa2c80d4fbdba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.452109 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 06c08679f1e746538bfa2c80d4fbdba3 (table default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:04.453210 1177 tablet_bootstrap.cc:654] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.453547 1266 raft_consensus.cc:493] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.453702 1266 raft_consensus.cc:515] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.454305 902 tablet_service.cc:1505] Processing CreateTablet for tablet 7792ee223dbe43aa82990541caced045 (DEFAULT_TABLE table=default.Table_e18fb157b5f64bbb951cc5ca8c5e3936 [id=335946e15b7c424ba3b769c46bd76ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.454416 1266 leader_election.cc:290] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.454702 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7792ee223dbe43aa82990541caced045. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.455817 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3a28a4aa361468c9fa9e72ccd6e7b7f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:04.456496 720 leader_election.cc:343] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.457197 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3a28a4aa361468c9fa9e72ccd6e7b7f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:04.457589 925 raft_consensus.cc:2468] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
W20251029 08:52:04.457765 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 24ed199b700f4f92858c4edb95217d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ed199b700f4f92858c4edb95217d03 already in progress: creating tablet
I20251029 08:52:04.458223 718 leader_election.cc:304] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
W20251029 08:52:04.458468 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a57b1dbbdf854bc6a6e1b5dc88b63bff consensus state has no leader
I20251029 08:52:04.458714 1266 raft_consensus.cc:2804] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:04.458820 1266 raft_consensus.cc:493] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:04.458931 1266 raft_consensus.cc:3060] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:04.461624 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:04.462360 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bfa1f140404f4768b6ae7c7fabc75d26" } modify_external_catalogs: true
I20251029 08:52:04.462546 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bfa1f140404f4768b6ae7c7fabc75d26" } modify_external_catalogs: true
I20251029 08:52:04.463299 1266 raft_consensus.cc:515] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.465097 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3a28a4aa361468c9fa9e72ccd6e7b7f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:04.465286 925 raft_consensus.cc:3060] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:04.465615 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3a28a4aa361468c9fa9e72ccd6e7b7f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:04.466178 720 leader_election.cc:343] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.466459 1266 leader_election.cc:290] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:04.467015 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 904523f7164849139cddb1100e95b824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904523f7164849139cddb1100e95b824 already in progress: creating tablet
I20251029 08:52:04.468564 925 raft_consensus.cc:2468] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:04.470356 718 leader_election.cc:304] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
W20251029 08:52:04.471460 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 25b2e13184de4c25a5f14bc78d871bda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25b2e13184de4c25a5f14bc78d871bda already in progress: creating tablet
I20251029 08:52:04.471717 1175 ts_tablet_manager.cc:1929] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.471843 1175 log.cc:1199] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4f65f8ca443e48a48d7ef4a474dc95b9
I20251029 08:52:04.471875 1176 tablet_replica.cc:333] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.472046 1266 raft_consensus.cc:2804] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:04.472304 1175 ts_tablet_manager.cc:1950] T 4f65f8ca443e48a48d7ef4a474dc95b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.472518 1176 raft_consensus.cc:2243] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.472699 1176 raft_consensus.cc:2272] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.473440 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b4ceeaba655542dab2556d3988b6e841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ceeaba655542dab2556d3988b6e841 already in progress: creating tablet
I20251029 08:52:04.474123 1176 ts_tablet_manager.cc:1916] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.476593 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4f65f8ca443e48a48d7ef4a474dc95b9 (table default.Table_7966c1de42494873aa075b0d9185824d [id=9b51cfa4d70542a280cc72c9528c45d4]) successfully deleted
I20251029 08:52:04.477062 1266 raft_consensus.cc:697] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:04.477514 1266 consensus_queue.cc:237] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:04.479296 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b55ae84c8d441459a5931abda471f6c consensus state has no leader
I20251029 08:52:04.480774 593 master_service.cc:954] ReplaceTablet: received request to replace tablet e899e183e255498b83a6d740af98de42 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.481030 1179 tablet_replica.cc:333] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.481323 1179 raft_consensus.cc:2243] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:04.481416 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_07a54a54b70a4f0aaea54ff93aa4bb65" } modify_external_catalogs: true
I20251029 08:52:04.481599 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_07a54a54b70a4f0aaea54ff93aa4bb65" } modify_external_catalogs: true
I20251029 08:52:04.481827 1179 pending_rounds.cc:70] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:04.481938 1179 pending_rounds.cc:77] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037576797614080 op_type: NO_OP noop_request { }
I20251029 08:52:04.482137 1179 raft_consensus.cc:2889] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:04.482311 1179 raft_consensus.cc:2272] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.483019 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cf4934c3842a463e857262ceed37e0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf4934c3842a463e857262ceed37e0bc already in progress: creating tablet
I20251029 08:52:04.484434 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:04.485635 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_effa05baab254c59bf4b358171091f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.486567 902 tablet_service.cc:1505] Processing CreateTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:04.486948 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_effa05baab254c59bf4b358171091f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.487056 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55842c18dbef4513a61dd9629f2e5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.488422 762 tablet_service.cc:1505] Processing CreateTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.489153 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55842c18dbef4513a61dd9629f2e5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.492231 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 06eb710809754e7abd00a345e8ae591a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06eb710809754e7abd00a345e8ae591a already in progress: creating tablet
I20251029 08:52:04.492573 1175 tablet_replica.cc:333] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.492683 1178 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.492811 1175 raft_consensus.cc:2243] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.492842 1178 ts_tablet_manager.cc:1403] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.003s
I20251029 08:52:04.492990 1175 raft_consensus.cc:2272] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.493659 1175 ts_tablet_manager.cc:1916] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.493706 1178 raft_consensus.cc:359] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.493907 1178 raft_consensus.cc:385] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.494005 1178 raft_consensus.cc:740] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.494257 1178 consensus_queue.cc:260] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.494796 1178 ts_tablet_manager.cc:1434] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:04.495132 1178 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.497355 1178 tablet_bootstrap.cc:654] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.498759 1177 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.498948 1177 ts_tablet_manager.cc:1403] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.006s
I20251029 08:52:04.499855 1177 raft_consensus.cc:359] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.500159 1177 raft_consensus.cc:385] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.500264 1177 raft_consensus.cc:740] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.500592 1177 consensus_queue.cc:260] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.501253 1177 ts_tablet_manager.cc:1434] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:04.507985 1177 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.509238 1175 ts_tablet_manager.cc:1929] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.509399 1175 log.cc:1199] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e899e183e255498b83a6d740af98de42
I20251029 08:52:04.509902 1175 ts_tablet_manager.cc:1950] T e899e183e255498b83a6d740af98de42 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.509898 1177 tablet_bootstrap.cc:654] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.512173 1176 ts_tablet_manager.cc:1929] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.516325 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e4cdf03acb92449d847aec897868b967
I20251029 08:52:04.516531 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e899e183e255498b83a6d740af98de42 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
I20251029 08:52:04.516868 1175 tablet_replica.cc:333] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.517084 1175 raft_consensus.cc:2243] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.518013 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:04.518086 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a7fdbe6681894ffa8264cdfa486f16e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7fdbe6681894ffa8264cdfa486f16e6 already in progress: creating tablet
I20251029 08:52:04.517856 1175 raft_consensus.cc:2272] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.518322 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 954c264e600a46f19db40dde5180787d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 954c264e600a46f19db40dde5180787d already in progress: creating tablet
W20251029 08:52:04.518584 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a2947083e3e447d5addae5effd186eb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2947083e3e447d5addae5effd186eb8 already in progress: creating tablet
I20251029 08:52:04.518939 1175 ts_tablet_manager.cc:1916] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.520753 1177 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.520891 1177 ts_tablet_manager.cc:1403] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20251029 08:52:04.521715 1177 raft_consensus.cc:359] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.521905 1177 raft_consensus.cc:385] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.522006 1177 raft_consensus.cc:740] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.522276 1177 consensus_queue.cc:260] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.522823 1177 ts_tablet_manager.cc:1434] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:04.523545 1177 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.484238 1043 tablet_service.cc:1505] Processing CreateTablet for tablet e687f593eead488db4bb445a63beeec7 (DEFAULT_TABLE table=default.Table_4b21589e83794d8e981ca8deeaf26b9f [id=a23a597c8fd343769b517507845104ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.517535 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.518954 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.524921 900 tablet_service.cc:1505] Processing CreateTablet for tablet e687f593eead488db4bb445a63beeec7 (DEFAULT_TABLE table=default.Table_4b21589e83794d8e981ca8deeaf26b9f [id=a23a597c8fd343769b517507845104ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.526468 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55842c18dbef4513a61dd9629f2e5a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.526849 1179 ts_tablet_manager.cc:1916] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.518395 762 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.526275 756 tablet_service.cc:1505] Processing CreateTablet for tablet 983a406aeed243f29ab322f94a0f5239 (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.530974 1176 log.cc:1199] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.531114 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983a406aeed243f29ab322f94a0f5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.531363 1176 ts_tablet_manager.cc:1950] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.532779 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
I20251029 08:52:04.533269 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e687f593eead488db4bb445a63beeec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.533924 1176 tablet_replica.cc:333] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.541061 1178 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.541285 1178 ts_tablet_manager.cc:1403] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.007s
I20251029 08:52:04.542212 1179 ts_tablet_manager.cc:1929] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:04.542152 1178 raft_consensus.cc:359] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.542362 1178 raft_consensus.cc:385] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.542441 1178 raft_consensus.cc:740] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.542696 1178 consensus_queue.cc:260] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.543573 1179 log.cc:1199] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/1557722a7c8041c88ee5321c0944e234
I20251029 08:52:04.543458 1178 ts_tablet_manager.cc:1434] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251029 08:52:04.544341 1179 ts_tablet_manager.cc:1950] T 1557722a7c8041c88ee5321c0944e234 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.544366 1178 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.545933 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 1557722a7c8041c88ee5321c0944e234 (table default.Table_3a8b8f91860e419e8538f028e57603fd [id=224c13af437a46fab3f2bd9b3d20f523]) successfully deleted
I20251029 08:52:04.546929 1179 tablet_replica.cc:333] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.547127 1179 raft_consensus.cc:2243] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.547322 1179 raft_consensus.cc:2272] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.548235 1176 raft_consensus.cc:2243] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:04.548735 1176 pending_rounds.cc:70] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:04.548813 1176 pending_rounds.cc:77] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037578663235584 op_type: NO_OP noop_request { }
I20251029 08:52:04.549062 1176 raft_consensus.cc:2889] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:04.549191 1176 raft_consensus.cc:2272] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.549530 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 54615.6us
I20251029 08:52:04.550431 1176 ts_tablet_manager.cc:1916] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.526226 1177 tablet_bootstrap.cc:654] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.527900 1175 ts_tablet_manager.cc:1929] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.551913 1175 log.cc:1199] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:04.552155 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e687f593eead488db4bb445a63beeec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.552418 1175 ts_tablet_manager.cc:1950] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.552663 1178 tablet_bootstrap.cc:654] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.553526 1175 tablet_replica.cc:333] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.553817 1175 raft_consensus.cc:2243] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.553833 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (table default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]) successfully deleted
I20251029 08:52:04.554072 1175 raft_consensus.cc:2272] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.554105 1266 raft_consensus.cc:493] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.554333 1266 raft_consensus.cc:515] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.554749 896 tablet_service.cc:1505] Processing CreateTablet for tablet 983a406aeed243f29ab322f94a0f5239 (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.555233 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983a406aeed243f29ab322f94a0f5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.555764 1266 leader_election.cc:290] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.556198 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453911c5a29f43e7a21fa2c47ef264e8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:04.556318 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453911c5a29f43e7a21fa2c47ef264e8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:04.557780 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bb2ad1c3e1ce4bebab6c5d2df73cd8fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.558372 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb2ad1c3e1ce4bebab6c5d2df73cd8fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.559265 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5288863701d147a28d9ee2a2d5e1754f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.560719 1175 ts_tablet_manager.cc:1916] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.560897 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1087b19fb3a74025b8269c7eb247bc14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.561050 1179 ts_tablet_manager.cc:1916] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:04.561342 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1087b19fb3a74025b8269c7eb247bc14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.562196 899 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:04.562255 718 leader_election.cc:343] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:04.562703 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5288863701d147a28d9ee2a2d5e1754f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:04.563103 720 leader_election.cc:343] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.563249 720 leader_election.cc:304] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:04.570921 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 983a406aeed243f29ab322f94a0f5239 (DEFAULT_TABLE table=default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.571089 1175 ts_tablet_manager.cc:1929] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.571254 1175 log.cc:1199] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/06c08679f1e746538bfa2c80d4fbdba3
I20251029 08:52:04.571460 1176 ts_tablet_manager.cc:1929] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:04.571614 1175 ts_tablet_manager.cc:1950] T 06c08679f1e746538bfa2c80d4fbdba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.571846 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983a406aeed243f29ab322f94a0f5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.572747 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 06c08679f1e746538bfa2c80d4fbdba3 (table default.Table_aa83a84a696b4573b7ceeb7a274a698e [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:04.572850 1183 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.573858 1183 ts_tablet_manager.cc:1403] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.215s user 0.008s sys 0.033s
I20251029 08:52:04.575320 1183 raft_consensus.cc:359] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.576838 1183 raft_consensus.cc:385] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.577112 1183 raft_consensus.cc:740] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.577634 1183 consensus_queue.cc:260] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.580039 1183 ts_tablet_manager.cc:1434] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:04.580506 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.580971 1183 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.581553 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dc5ebeb36043e0aa5955bacd06c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.584134 1177 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.584326 1177 ts_tablet_manager.cc:1403] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.002s
I20251029 08:52:04.585144 1177 raft_consensus.cc:359] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.585359 1177 raft_consensus.cc:385] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.585438 1177 raft_consensus.cc:740] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.585430 1179 ts_tablet_manager.cc:1929] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.585553 1179 log.cc:1199] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a232a4a898f04f339084c807990a14e1
I20251029 08:52:04.585708 1177 consensus_queue.cc:260] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.587124 896 tablet_service.cc:1505] Processing CreateTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.587567 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dc5ebeb36043e0aa5955bacd06c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.586232 1177 ts_tablet_manager.cc:1434] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:04.588122 1177 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.587338 1179 ts_tablet_manager.cc:1950] T a232a4a898f04f339084c807990a14e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.591532 1175 tablet_replica.cc:333] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.591725 1175 raft_consensus.cc:2243] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.591907 1175 raft_consensus.cc:2272] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.577221 756 tablet_service.cc:1505] Processing CreateTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef (DEFAULT_TABLE table=default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.592581 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dc5ebeb36043e0aa5955bacd06c2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.593053 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a232a4a898f04f339084c807990a14e1 (table default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]) successfully deleted
W20251029 08:52:04.594007 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f consensus state has no leader
I20251029 08:52:04.573630 1266 raft_consensus.cc:2749] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:04.598739 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 06eb710809754e7abd00a345e8ae591a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06eb710809754e7abd00a345e8ae591a already in progress: creating tablet
I20251029 08:52:04.571606 1176 log.cc:1199] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:04.600524 1176 ts_tablet_manager.cc:1950] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:04.601305 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a7fdbe6681894ffa8264cdfa486f16e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7fdbe6681894ffa8264cdfa486f16e6 already in progress: creating tablet
I20251029 08:52:04.592005 1177 tablet_bootstrap.cc:654] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.592443 1175 ts_tablet_manager.cc:1916] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.601573 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (table default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]) successfully deleted
I20251029 08:52:04.601586 1179 tablet_replica.cc:333] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.601876 1266 raft_consensus.cc:493] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.602027 1179 raft_consensus.cc:2243] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.602073 1266 raft_consensus.cc:515] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.602509 1179 raft_consensus.cc:2272] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.603035 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:04.603243 925 raft_consensus.cc:2468] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:04.603547 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:04.603919 1179 ts_tablet_manager.cc:1916] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.604127 718 leader_election.cc:304] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:04.604403 1323 raft_consensus.cc:2804] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:04.604497 1323 raft_consensus.cc:493] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:04.604597 1323 raft_consensus.cc:3060] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:04.604609 720 leader_election.cc:343] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.604795 1266 leader_election.cc:290] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.605461 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.606002 1176 tablet_replica.cc:333] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.606613 1176 raft_consensus.cc:2243] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.606803 1176 raft_consensus.cc:2272] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.607995 1323 raft_consensus.cc:515] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.608976 1323 leader_election.cc:290] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.609493 1175 ts_tablet_manager.cc:1929] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.609619 1175 log.cc:1199] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.609982 1175 ts_tablet_manager.cc:1950] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.610468 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:04.610644 925 raft_consensus.cc:3060] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:04.610967 1176 ts_tablet_manager.cc:1916] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:04.612677 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4deeec492a2545ca86583c6ea899a0ab consensus state has no leader
I20251029 08:52:04.614554 925 raft_consensus.cc:2468] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:04.615011 718 leader_election.cc:304] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:04.615356 1323 raft_consensus.cc:2804] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:04.615669 1323 raft_consensus.cc:697] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:04.590780 1183 tablet_bootstrap.cc:654] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.610056 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:04.616350 720 leader_election.cc:343] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.618000 1323 consensus_queue.cc:237] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.619060 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
I20251029 08:52:04.621026 1176 ts_tablet_manager.cc:1929] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.621158 1176 log.cc:1199] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/51bb01c123c44fb69b2f174fca5d7a13
I20251029 08:52:04.621569 1176 ts_tablet_manager.cc:1950] T 51bb01c123c44fb69b2f174fca5d7a13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.623363 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a61235da8c7a4f31b29d5d50c53c347d
I20251029 08:52:04.623483 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4cf906c73dc84403bcc3dece74a822bd
I20251029 08:52:04.624047 591 catalog_manager.cc:6719] ReplaceTablet: tablet 4cf906c73dc84403bcc3dece74a822bd deleted and replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464
I20251029 08:52:04.624258 1175 tablet_replica.cc:333] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.624382 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec9165a4204e4f209e0a28ea2e0c04fe" } modify_external_catalogs: true
I20251029 08:52:04.624481 1175 raft_consensus.cc:2243] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.624552 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec9165a4204e4f209e0a28ea2e0c04fe" } modify_external_catalogs: true
I20251029 08:52:04.624651 1175 raft_consensus.cc:2272] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.625061 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:04.625906 1175 ts_tablet_manager.cc:1916] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.626121 596 catalog_manager.cc:6719] ReplaceTablet: tablet 453911c5a29f43e7a21fa2c47ef264e8 deleted and replaced by tablet 6acb2a70edd044a289c4e4fd7df9643e
I20251029 08:52:04.626299 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 64f73597d1764540bcaf472d3e890ee8 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.626324 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.626550 762 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.627468 896 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.627794 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6acb2a70edd044a289c4e4fd7df9643e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.628527 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dae092a6bac84104903dcf04163fc4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.629133 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dae092a6bac84104903dcf04163fc4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.629993 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6acb2a70edd044a289c4e4fd7df9643e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.630468 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.630589 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6acb2a70edd044a289c4e4fd7df9643e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.631743 1183 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.631942 1183 ts_tablet_manager.cc:1403] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.001s
I20251029 08:52:04.632287 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9869b4e9e33342c4bc5fe57a67b675ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.632895 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9869b4e9e33342c4bc5fe57a67b675ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.633030 1183 raft_consensus.cc:359] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.633301 1183 raft_consensus.cc:385] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.633441 1183 raft_consensus.cc:740] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.633716 1183 consensus_queue.cc:260] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.634513 1175 ts_tablet_manager.cc:1929] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.634642 1175 log.cc:1199] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:04.634622 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 51bb01c123c44fb69b2f174fca5d7a13 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
I20251029 08:52:04.635010 1175 ts_tablet_manager.cc:1950] T db9cf2a8f2254a38a725bcd3c83e8509 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.635345 1183 ts_tablet_manager.cc:1434] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:04.635705 1183 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:04.635766 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e4cdf03acb92449d847aec897868b967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cdf03acb92449d847aec897868b967 already in progress: creating tablet
W20251029 08:52:04.637152 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4cf906c73dc84403bcc3dece74a822bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cf906c73dc84403bcc3dece74a822bd already in progress: creating tablet
I20251029 08:52:04.637492 1176 tablet_replica.cc:333] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:04.637617 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a61235da8c7a4f31b29d5d50c53c347d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a61235da8c7a4f31b29d5d50c53c347d already in progress: creating tablet
I20251029 08:52:04.637777 1177 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.637948 1177 ts_tablet_manager.cc:1403] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.007s
I20251029 08:52:04.638180 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.638232 1176 raft_consensus.cc:2243] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.638404 1176 raft_consensus.cc:2272] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.638873 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.639118 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.638876 1177 raft_consensus.cc:359] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.639400 1177 raft_consensus.cc:385] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.639745 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.640201 1176 ts_tablet_manager.cc:1916] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.640348 1177 raft_consensus.cc:740] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.641069 1183 tablet_bootstrap.cc:654] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.641538 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet db9cf2a8f2254a38a725bcd3c83e8509 (table default.Table_cd11e136b8054bdfb9c12527b0fc5777 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
W20251029 08:52:04.641779 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e4cdf03acb92449d847aec897868b967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cdf03acb92449d847aec897868b967 already in progress: creating tablet
I20251029 08:52:04.642234 1177 consensus_queue.cc:260] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.641536 1175 tablet_replica.cc:333] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.642768 1175 raft_consensus.cc:2243] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.642966 1175 raft_consensus.cc:2272] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.643419 1177 ts_tablet_manager.cc:1434] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 08:52:04.645524 1178 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.645658 1178 ts_tablet_manager.cc:1403] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.010s
I20251029 08:52:04.647902 1177 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.643755 1175 ts_tablet_manager.cc:1916] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.650168 1178 raft_consensus.cc:359] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.650385 1178 raft_consensus.cc:385] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.650486 1178 raft_consensus.cc:740] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.650847 1178 consensus_queue.cc:260] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.652953 1178 ts_tablet_manager.cc:1434] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:04.654620 1183 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.655483 1177 tablet_bootstrap.cc:654] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.655803 1183 ts_tablet_manager.cc:1403] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20251029 08:52:04.656843 1183 raft_consensus.cc:359] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.657784 1178 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.659534 1178 tablet_bootstrap.cc:654] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.659593 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b7aebe05237b4357bab8470b1e161bc0" } modify_external_catalogs: true
I20251029 08:52:04.659813 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b7aebe05237b4357bab8470b1e161bc0" } modify_external_catalogs: true
I20251029 08:52:04.660588 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.663300 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ca7b76be24b04d34b006fef05f3df689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.663894 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca7b76be24b04d34b006fef05f3df689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.642488 1179 ts_tablet_manager.cc:1929] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.666591 1183 raft_consensus.cc:385] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.666983 1175 ts_tablet_manager.cc:1929] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.667126 1175 log.cc:1199] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/be476c9994ed4dc28f606955b16bc4de
I20251029 08:52:04.666697 1179 log.cc:1199] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0d8a59f8f764466686a4b25c3b28596f
I20251029 08:52:04.667531 1175 ts_tablet_manager.cc:1950] T be476c9994ed4dc28f606955b16bc4de P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.667562 1179 ts_tablet_manager.cc:1950] T 0d8a59f8f764466686a4b25c3b28596f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.668576 1175 tablet_replica.cc:333] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.668706 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0d8a59f8f764466686a4b25c3b28596f (table default.Table_a2cb1179d9de4d818e95e90c74afef35 [id=cf4fd3de658d4e63920fd03231a3e553]) successfully deleted
I20251029 08:52:04.668782 1175 raft_consensus.cc:2243] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.668772 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet be476c9994ed4dc28f606955b16bc4de (table default.Table_1c46023036d34ee7be5bb30a14caa0e2 [id=6c75c2c51a93469489dfd59bce7906c6]) successfully deleted
I20251029 08:52:04.668946 1175 raft_consensus.cc:2272] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.669512 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a2947083e3e447d5addae5effd186eb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2947083e3e447d5addae5effd186eb8 already in progress: creating tablet
I20251029 08:52:04.669867 1179 tablet_replica.cc:333] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.669984 1176 ts_tablet_manager.cc:1929] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.670120 1179 raft_consensus.cc:2243] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.670137 1176 log.cc:1199] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:04.670368 1179 raft_consensus.cc:2272] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.670514 1176 ts_tablet_manager.cc:1950] T 453911c5a29f43e7a21fa2c47ef264e8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.670632 1183 raft_consensus.cc:740] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.671180 1183 consensus_queue.cc:260] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.672694 590 master_service.cc:954] ReplaceTablet: received request to replace tablet cf4934c3842a463e857262ceed37e0bc from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.673290 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_29ca0b751907426496780bf44111f5d2" } new_table_name: "default.Table_bf69c5ad108f412f8a742d83448bd6e3" modify_external_catalogs: true
I20251029 08:52:04.673516 1183 ts_tablet_manager.cc:1434] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251029 08:52:04.673816 1183 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.675510 1179 ts_tablet_manager.cc:1916] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.677834 1178 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.678015 1178 ts_tablet_manager.cc:1403] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251029 08:52:04.679454 1178 raft_consensus.cc:359] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.679653 1178 raft_consensus.cc:385] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.679737 1178 raft_consensus.cc:740] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.680509 1178 consensus_queue.cc:260] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.681934 1178 ts_tablet_manager.cc:1434] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20251029 08:52:04.682518 1175 ts_tablet_manager.cc:1916] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.683080 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 64f73597d1764540bcaf472d3e890ee8
I20251029 08:52:04.683588 596 catalog_manager.cc:6719] ReplaceTablet: tablet 64f73597d1764540bcaf472d3e890ee8 deleted and replaced by tablet 3295bb0b47714080a5c3f34049acd487
I20251029 08:52:04.683977 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.683534 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 453911c5a29f43e7a21fa2c47ef264e8 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
I20251029 08:52:04.684823 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.685144 1178 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.685609 1183 tablet_bootstrap.cc:654] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.686067 1177 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.686168 1266 raft_consensus.cc:493] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.686362 1266 raft_consensus.cc:515] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.686966 1266 leader_election.cc:290] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:04.687568 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 64f73597d1764540bcaf472d3e890ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f73597d1764540bcaf472d3e890ee8 already in progress: creating tablet
I20251029 08:52:04.687804 1177 ts_tablet_manager.cc:1403] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.002s
I20251029 08:52:04.688038 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.688365 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:04.689003 1178 tablet_bootstrap.cc:654] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:04.689111 718 leader_election.cc:343] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.689538 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e18fb157b5f64bbb951cc5ca8c5e3936" } new_table_name: "default.Table_9b80aae43bb845d1bb98687e0e6116f6" modify_external_catalogs: true
I20251029 08:52:04.691300 1177 raft_consensus.cc:359] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.691519 1177 raft_consensus.cc:385] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.691596 1177 raft_consensus.cc:740] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.691823 1177 consensus_queue.cc:260] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.687467 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:04.692706 720 leader_election.cc:343] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.692857 720 leader_election.cc:304] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:04.693375 1323 raft_consensus.cc:2749] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:04.694758 1177 ts_tablet_manager.cc:1434] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 08:52:04.690435 1179 ts_tablet_manager.cc:1929] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.696084 1177 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.697840 1177 tablet_bootstrap.cc:654] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.698009 1179 log.cc:1199] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/954c264e600a46f19db40dde5180787d
I20251029 08:52:04.698457 1179 ts_tablet_manager.cc:1950] T 954c264e600a46f19db40dde5180787d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.699606 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 954c264e600a46f19db40dde5180787d (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
W20251029 08:52:04.700347 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3a28a4aa361468c9fa9e72ccd6e7b7f already in progress: creating tablet
I20251029 08:52:04.700651 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6acb2a70edd044a289c4e4fd7df9643e
I20251029 08:52:04.701503 1179 tablet_replica.cc:333] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.703442 762 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.703835 1176 tablet_replica.cc:333] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.703889 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.704151 1176 raft_consensus.cc:2243] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.704411 1176 raft_consensus.cc:2272] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.705569 1176 ts_tablet_manager.cc:1916] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.706808 1177 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.706986 1177 ts_tablet_manager.cc:1403] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251029 08:52:04.707808 1177 raft_consensus.cc:359] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.708005 1177 raft_consensus.cc:385] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.708148 1177 raft_consensus.cc:740] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.708523 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0567a7bfaea44a6d8bd8807e6206939c" } new_table_name: "default.Table_2e15dbeda3ab4c4b8473ccd12597b270" modify_external_catalogs: true
I20251029 08:52:04.710301 1177 consensus_queue.cc:260] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.710887 1177 ts_tablet_manager.cc:1434] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:04.712719 1177 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.713176 1179 raft_consensus.cc:2243] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.713402 1179 raft_consensus.cc:2272] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.716972 1176 ts_tablet_manager.cc:1929] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.717669 1179 ts_tablet_manager.cc:1916] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.718605 1176 log.cc:1199] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/cc1d1d5b2a9e4051bbcd25be1578ee73
I20251029 08:52:04.719040 1176 ts_tablet_manager.cc:1950] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:04.718472 1178 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.719306 1178 ts_tablet_manager.cc:1403] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.005s
I20251029 08:52:04.720209 1178 raft_consensus.cc:359] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.720460 1178 raft_consensus.cc:385] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.720602 1178 raft_consensus.cc:740] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.720896 1178 consensus_queue.cc:260] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.721625 1178 ts_tablet_manager.cc:1434] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:04.721952 1178 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.723779 1178 tablet_bootstrap.cc:654] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.724807 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (table default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]) successfully deleted
W20251029 08:52:04.725800 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 453911c5a29f43e7a21fa2c47ef264e8 because the tablet was not found. No further retry: Not found: Tablet not found: 453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:04.729178 1175 ts_tablet_manager.cc:1929] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.729326 1175 log.cc:1199] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/06eb710809754e7abd00a345e8ae591a
I20251029 08:52:04.729658 1175 ts_tablet_manager.cc:1950] T 06eb710809754e7abd00a345e8ae591a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.730994 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 06eb710809754e7abd00a345e8ae591a (table default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]) successfully deleted
I20251029 08:52:04.733026 1177 tablet_bootstrap.cc:654] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.733901 1183 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.734133 1183 ts_tablet_manager.cc:1403] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.007s
I20251029 08:52:04.734263 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b58dc38f334840779d240a10ccb0d8db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.734881 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a61235da8c7a4f31b29d5d50c53c347d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a61235da8c7a4f31b29d5d50c53c347d already in progress: creating tablet
W20251029 08:52:04.735016 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b58dc38f334840779d240a10ccb0d8db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.735296 1178 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.735605 1175 tablet_replica.cc:333] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.735452 1178 ts_tablet_manager.cc:1403] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20251029 08:52:04.735778 1175 raft_consensus.cc:2243] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.735934 1175 raft_consensus.cc:2272] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.736555 1178 raft_consensus.cc:359] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.736760 1178 raft_consensus.cc:385] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.736850 1178 raft_consensus.cc:740] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.737099 1178 consensus_queue.cc:260] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.737739 1178 ts_tablet_manager.cc:1434] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:04.738195 1175 ts_tablet_manager.cc:1916] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.739022 1177 tablet_bootstrap.cc:492] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.739185 1177 ts_tablet_manager.cc:1403] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251029 08:52:04.740258 1178 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.741290 1177 raft_consensus.cc:359] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.741503 1177 raft_consensus.cc:385] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.741575 1177 raft_consensus.cc:740] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.741799 1177 consensus_queue.cc:260] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.741989 597 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6138208f0f6f415bbda28f2ec61cba49
I20251029 08:52:04.742278 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.742321 1177 ts_tablet_manager.cc:1434] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:04.742666 1177 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.743355 1238 raft_consensus.cc:493] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.743570 1238 raft_consensus.cc:515] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.744060 1178 tablet_bootstrap.cc:654] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.743983 1183 raft_consensus.cc:359] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.745772 1183 raft_consensus.cc:385] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.745867 1183 raft_consensus.cc:740] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.746094 1183 consensus_queue.cc:260] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:04.746326 998 leader_election.cc:343] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.746526 1179 ts_tablet_manager.cc:1929] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.746707 1179 log.cc:1199] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/8ded7b3943e544e585fe52433ff7555f
I20251029 08:52:04.747138 1179 ts_tablet_manager.cc:1950] T 8ded7b3943e544e585fe52433ff7555f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.748196 1179 tablet_replica.cc:333] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.748301 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 8ded7b3943e544e585fe52433ff7555f (table default.Table_389d5f5c55ce46d5ab8a5729cbe1d3b3 [id=5124fdf717764856bff708bedd9656c8]) successfully deleted
I20251029 08:52:04.748432 1179 raft_consensus.cc:2243] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.748596 1179 raft_consensus.cc:2272] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.752449 996 leader_election.cc:343] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.752619 996 leader_election.cc:304] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:04.752777 1177 tablet_bootstrap.cc:654] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:04.752843 996 raft_consensus.cc:2686] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:04.744386 1238 leader_election.cc:290] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:04.754505 1179 ts_tablet_manager.cc:1916] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.768993 1175 ts_tablet_manager.cc:1929] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.769165 1175 log.cc:1199] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/6b8a4b96d3c748daa0f467768b2b7abd
I20251029 08:52:04.770885 1179 ts_tablet_manager.cc:1929] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.772426 1179 log.cc:1199] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3c9a18c9e63d4b5e9a42fe053b187b08
I20251029 08:52:04.773021 1179 ts_tablet_manager.cc:1950] T 3c9a18c9e63d4b5e9a42fe053b187b08 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.773800 1177 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.773793 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d65e71e019114f57877994e1b43540ef" } modify_external_catalogs: true
I20251029 08:52:04.774011 1177 ts_tablet_manager.cc:1403] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20251029 08:52:04.774009 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d65e71e019114f57877994e1b43540ef" } modify_external_catalogs: true
I20251029 08:52:04.775309 1177 raft_consensus.cc:359] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.775501 1177 raft_consensus.cc:385] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.775573 1177 raft_consensus.cc:740] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.775830 1177 consensus_queue.cc:260] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.776492 1177 ts_tablet_manager.cc:1434] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:04.777264 1177 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.780987 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3c9a18c9e63d4b5e9a42fe053b187b08 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
W20251029 08:52:04.782325 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3295bb0b47714080a5c3f34049acd487 has no consensus state
W20251029 08:52:04.782855 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e4cdf03acb92449d847aec897868b967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cdf03acb92449d847aec897868b967 already in progress: creating tablet
I20251029 08:52:04.783707 1179 tablet_replica.cc:333] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:04.782238 1177 tablet_bootstrap.cc:654] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.784570 1179 raft_consensus.cc:2243] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.784818 1179 raft_consensus.cc:2272] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.780350 1175 ts_tablet_manager.cc:1950] T 6b8a4b96d3c748daa0f467768b2b7abd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.786342 1177 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:04.786480 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 6b8a4b96d3c748daa0f467768b2b7abd (table default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]) successfully deleted
I20251029 08:52:04.786509 1177 ts_tablet_manager.cc:1403] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251029 08:52:04.786547 1175 tablet_replica.cc:333] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:04.786764 1175 raft_consensus.cc:2243] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.786758 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3d66a4f6f5af4c4688c532261fc9ccfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.786936 1175 raft_consensus.cc:2272] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:04.787099 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4cf906c73dc84403bcc3dece74a822bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cf906c73dc84403bcc3dece74a822bd already in progress: creating tablet
W20251029 08:52:04.787380 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d66a4f6f5af4c4688c532261fc9ccfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.787323 1177 raft_consensus.cc:359] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.787518 1177 raft_consensus.cc:385] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.787588 1177 raft_consensus.cc:740] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.786185 1179 ts_tablet_manager.cc:1916] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:04.787818 1177 consensus_queue.cc:260] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.789983 1177 ts_tablet_manager.cc:1434] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:04.790838 1177 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:04.793035 1175 ts_tablet_manager.cc:1916] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:04.795049 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7792ee223dbe43aa82990541caced045 consensus state has no leader
I20251029 08:52:04.797327 1179 ts_tablet_manager.cc:1929] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.797526 1179 log.cc:1199] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/be476c9994ed4dc28f606955b16bc4de
I20251029 08:52:04.797899 1179 ts_tablet_manager.cc:1950] T be476c9994ed4dc28f606955b16bc4de P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:04.798918 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet be476c9994ed4dc28f606955b16bc4de (table default.Table_1c46023036d34ee7be5bb30a14caa0e2 [id=6c75c2c51a93469489dfd59bce7906c6]) successfully deleted
I20251029 08:52:04.799343 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6d2b9c809be24c519e176adfd43dfa45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.799604 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4cf906c73dc84403bcc3dece74a822bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cf906c73dc84403bcc3dece74a822bd already in progress: creating tablet
W20251029 08:52:04.800076 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d2b9c809be24c519e176adfd43dfa45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:04.800455 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a61235da8c7a4f31b29d5d50c53c347d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a61235da8c7a4f31b29d5d50c53c347d already in progress: creating tablet
W20251029 08:52:04.801124 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 64f73597d1764540bcaf472d3e890ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f73597d1764540bcaf472d3e890ee8 already in progress: creating tablet
I20251029 08:52:04.803187 1183 ts_tablet_manager.cc:1434] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.069s user 0.004s sys 0.012s
I20251029 08:52:04.803593 1183 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.806295 1183 tablet_bootstrap.cc:654] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:04.808171 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13eeb2e8951b421e9d6f8f228a956e3d consensus state has no leader
I20251029 08:52:04.811991 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cb01810d231f447986917d7d5d1276b7" } new_table_name: "default.Table_8325dc8d37f147e79a2672a42a930de9" modify_external_catalogs: true
I20251029 08:52:04.812310 1183 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.812528 1183 ts_tablet_manager.cc:1403] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20251029 08:52:04.813369 1183 raft_consensus.cc:359] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.813625 1183 raft_consensus.cc:385] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.813735 1183 raft_consensus.cc:740] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.814002 1183 consensus_queue.cc:260] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.814505 1183 ts_tablet_manager.cc:1434] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:04.814869 1183 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.817485 1183 tablet_bootstrap.cc:654] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.822811 1183 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.823081 1183 ts_tablet_manager.cc:1403] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20251029 08:52:04.824535 1183 raft_consensus.cc:359] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.824795 1183 raft_consensus.cc:385] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.824936 1183 raft_consensus.cc:740] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.825251 1183 consensus_queue.cc:260] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.826707 1183 ts_tablet_manager.cc:1434] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:04.828120 1183 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.832783 1183 tablet_bootstrap.cc:654] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.834587 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bf69c5ad108f412f8a742d83448bd6e3" } new_table_name: "default.Table_32228c67a14f44429dd7a7bbed1e1e34" modify_external_catalogs: true
I20251029 08:52:04.840816 1183 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.841172 1183 ts_tablet_manager.cc:1403] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20251029 08:52:04.841738 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]). Expected version 2 got 0
I20251029 08:52:04.842007 1183 raft_consensus.cc:359] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.842339 1183 raft_consensus.cc:385] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.842458 1183 raft_consensus.cc:740] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:04.842267 1262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 08:52:04.842705 1262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:52:04.842839 1262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251029 08:52:04.842831 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_742704efe2d14569b8a72c8e7d09c736" } modify_external_catalogs: true
I20251029 08:52:04.842846 1183 consensus_queue.cc:260] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.843037 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_742704efe2d14569b8a72c8e7d09c736" } modify_external_catalogs: true
I20251029 08:52:04.843730 1183 ts_tablet_manager.cc:1434] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:04.843930 595 catalog_manager.cc:5649] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:04.844292 1183 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.844460 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:04.849793 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 9702a7d8b8304b49a250c129ecf28303 (DEFAULT_TABLE table=default.Table_e735241cb8a04d689487db8e497b2d31 [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.849975 1183 tablet_bootstrap.cc:654] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.850256 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9702a7d8b8304b49a250c129ecf28303. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.851756 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e0b459b366cb402db2e7e8fee2b3e855 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0b459b366cb402db2e7e8fee2b3e855 consensus state has no leader
I20251029 08:52:04.854516 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.854929 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b534aaa51b4fad9dc3fc915de947f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.859895 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9702a7d8b8304b49a250c129ecf28303 (DEFAULT_TABLE table=default.Table_e735241cb8a04d689487db8e497b2d31 [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.860548 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9702a7d8b8304b49a250c129ecf28303. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.860582 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9702a7d8b8304b49a250c129ecf28303 (DEFAULT_TABLE table=default.Table_e735241cb8a04d689487db8e497b2d31 [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:04.861598 1262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 08:52:04.861675 1262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 08:52:04.861752 1262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 08:52:04.861797 1262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 08:52:04.862649 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9702a7d8b8304b49a250c129ecf28303. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.864766 1193 raft_consensus.cc:493] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.865706 762 tablet_service.cc:1505] Processing CreateTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.866132 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b534aaa51b4fad9dc3fc915de947f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.874477 1193 raft_consensus.cc:515] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.875314 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b0d5f8da41ce40019fd3c17f32f07b01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.875939 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0d5f8da41ce40019fd3c17f32f07b01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.878154 1193 leader_election.cc:290] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:04.879660 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e2c32b84201470aa0dd697a5da12117" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:04.879874 1071 raft_consensus.cc:2468] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:04.881466 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e2c32b84201470aa0dd697a5da12117" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:04.881774 1178 tablet_bootstrap.cc:492] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.881942 1178 ts_tablet_manager.cc:1403] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.011s
I20251029 08:52:04.882758 1178 raft_consensus.cc:359] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.882934 1178 raft_consensus.cc:385] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.883016 1178 raft_consensus.cc:740] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.883247 1178 consensus_queue.cc:260] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.886783 1183 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:04.886981 1183 ts_tablet_manager.cc:1403] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.002s
I20251029 08:52:04.887506 1178 ts_tablet_manager.cc:1434] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:04.887797 1178 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.890080 1178 tablet_bootstrap.cc:654] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.890252 861 leader_election.cc:304] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:04.892696 861 leader_election.cc:343] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.893129 1175 ts_tablet_manager.cc:1929] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:04.893548 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 63b08840d0fa40428c61a05b277ad750 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.893976 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b08840d0fa40428c61a05b277ad750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.894264 756 tablet_service.cc:1505] Processing CreateTablet for tablet 63b08840d0fa40428c61a05b277ad750 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.894647 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b08840d0fa40428c61a05b277ad750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.895298 1193 raft_consensus.cc:2804] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:04.895285 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf (DEFAULT_TABLE table=default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.895398 1193 raft_consensus.cc:493] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:04.895664 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9165c4bf484a48b4a0b7246cb11571bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.900108 1193 raft_consensus.cc:3060] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:04.900035 1177 tablet_bootstrap.cc:654] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.901950 1175 log.cc:1199] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:04.902328 1175 ts_tablet_manager.cc:1950] T 25b2e13184de4c25a5f14bc78d871bda P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:04.902259 1262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=37745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
I20251029 08:52:04.903692 1183 raft_consensus.cc:359] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.903888 1183 raft_consensus.cc:385] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.903981 1183 raft_consensus.cc:740] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.904528 1262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:52:04.904969 1183 consensus_queue.cc:260] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.905727 1183 ts_tablet_manager.cc:1434] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251029 08:52:04.906514 1183 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:04.907724 1262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 08:52:04.908336 1183 tablet_bootstrap.cc:654] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.910851 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf (DEFAULT_TABLE table=default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.911320 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9165c4bf484a48b4a0b7246cb11571bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.911844 596 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]). Expected version 2 got 0
I20251029 08:52:04.913861 900 tablet_service.cc:1505] Processing CreateTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.914615 902 tablet_service.cc:1505] Processing CreateTablet for tablet 63b08840d0fa40428c61a05b277ad750 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.915253 896 tablet_service.cc:1505] Processing CreateTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf (DEFAULT_TABLE table=default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.916009 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 25b2e13184de4c25a5f14bc78d871bda (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
W20251029 08:52:04.916298 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 64f73597d1764540bcaf472d3e890ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f73597d1764540bcaf472d3e890ee8 already in progress: creating tablet
W20251029 08:52:04.917695 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d64b1a762f124195b4369c096587881d consensus state has no leader
I20251029 08:52:04.917697 1193 raft_consensus.cc:515] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.918407 1193 leader_election.cc:290] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:04.918577 1193 raft_consensus.cc:493] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.918723 1193 raft_consensus.cc:515] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.919207 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_90db61e4d27e4b409e948b3f33c765e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.919713 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90db61e4d27e4b409e948b3f33c765e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.920625 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14adc0f009d94287b92eda7b171c7c13" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:04.920885 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e2c32b84201470aa0dd697a5da12117" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:04.923633 762 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 (DEFAULT_TABLE table=default.Table_4b89cba8e4a642dabf87ae4f9a509895 [id=13d97eefb49f42cc94cb040ed86f4aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.924161 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc95257c937454aa6cd55b5a9394a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.927620 749 tablet_service.cc:1505] Processing CreateTablet for tablet 28face87091b49d2971217618c95d2f5 (DEFAULT_TABLE table=default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.928078 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28face87091b49d2971217618c95d2f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.928968 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3295bb0b47714080a5c3f34049acd487 has no consensus state
I20251029 08:52:04.929342 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_25994a51dae84fef8fcd0519119cf2af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.929844 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25994a51dae84fef8fcd0519119cf2af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.932334 1193 leader_election.cc:290] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:04.935393 760 tablet_service.cc:1505] Processing CreateTablet for tablet 10baa0547180493f872a23614cf4c608 (DEFAULT_TABLE table=default.Table_971780f9a8d5435a9f6a94d5727f5119 [id=56e0b03feb7245d59f0f2ef3a8c977f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.935804 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10baa0547180493f872a23614cf4c608. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.938761 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b534aaa51b4fad9dc3fc915de947f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.939152 861 leader_election.cc:343] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.939774 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14adc0f009d94287b92eda7b171c7c13" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:04.946346 899 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 (DEFAULT_TABLE table=default.Table_4b89cba8e4a642dabf87ae4f9a509895 [id=13d97eefb49f42cc94cb040ed86f4aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.948890 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:04.950078 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:04.950733 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 55842c18dbef4513a61dd9629f2e5a4c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:04.953315 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:04.953879 597 catalog_manager.cc:6719] ReplaceTablet: tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f deleted and replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123
I20251029 08:52:04.955196 1178 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:04.955376 1178 ts_tablet_manager.cc:1403] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20251029 08:52:04.955730 1069 raft_consensus.cc:2468] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:04.956214 1178 raft_consensus.cc:359] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.956429 1178 raft_consensus.cc:385] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:04.956499 1178 raft_consensus.cc:740] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:04.958092 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e2c32b84201470aa0dd697a5da12117" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:04.958369 1071 raft_consensus.cc:3060] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:04.958408 1178 consensus_queue.cc:260] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.960008 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.960177 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.961232 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 28face87091b49d2971217618c95d2f5 (DEFAULT_TABLE table=default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.959996 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:04.961560 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7cbce5d595364b4bad36d59fb532c237 because the tablet was not found. No further retry: Not found: Tablet not found: 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:04.961773 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28face87091b49d2971217618c95d2f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.961799 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f already in progress: creating tablet
W20251029 08:52:04.962028 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 because the tablet was not found. No further retry: Not found: Tablet not found: 2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:04.962229 1178 ts_tablet_manager.cc:1434] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:04.962558 1178 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:04.964972 760 tablet_service.cc:1505] Processing CreateTablet for tablet 7cbce5d595364b4bad36d59fb532c237 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.965360 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbce5d595364b4bad36d59fb532c237. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.966131 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b08840d0fa40428c61a05b277ad750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.966411 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.966684 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.966849 905 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.967106 1071 raft_consensus.cc:2468] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:04.967723 861 leader_election.cc:343] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:04.967979 861 leader_election.cc:304] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:04.970240 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9165c4bf484a48b4a0b7246cb11571bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.970816 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.971256 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.971549 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:04.972344 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 10baa0547180493f872a23614cf4c608 (DEFAULT_TABLE table=default.Table_971780f9a8d5435a9f6a94d5727f5119 [id=56e0b03feb7245d59f0f2ef3a8c977f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.972755 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10baa0547180493f872a23614cf4c608. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.972822 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.973081 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.973192 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 7cbce5d595364b4bad36d59fb532c237 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.973340 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.973593 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbce5d595364b4bad36d59fb532c237. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:04.974045 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7cbce5d595364b4bad36d59fb532c237 because the tablet was not found. No further retry: Not found: Tablet not found: 7cbce5d595364b4bad36d59fb532c237
W20251029 08:52:04.974340 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet be476c9994ed4dc28f606955b16bc4de because the tablet was not found. No further retry: Not found: Tablet not found: be476c9994ed4dc28f606955b16bc4de
W20251029 08:52:04.974625 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 954c264e600a46f19db40dde5180787d because the tablet was not found. No further retry: Not found: Tablet not found: 954c264e600a46f19db40dde5180787d
I20251029 08:52:04.976158 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.976389 1323 raft_consensus.cc:493] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.976538 1323 raft_consensus.cc:515] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:04.977152 1323 leader_election.cc:290] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:04.973605 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:04.981490 1178 tablet_bootstrap.cc:654] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:04.981721 1193 raft_consensus.cc:493] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:04.982372 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc95257c937454aa6cd55b5a9394a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.971935 898 tablet_service.cc:1505] Processing CreateTablet for tablet 28face87091b49d2971217618c95d2f5 (DEFAULT_TABLE table=default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.983860 895 tablet_service.cc:1505] Processing CreateTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (DEFAULT_TABLE table=default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.984622 904 tablet_service.cc:1505] Processing CreateTablet for tablet 10baa0547180493f872a23614cf4c608 (DEFAULT_TABLE table=default.Table_971780f9a8d5435a9f6a94d5727f5119 [id=56e0b03feb7245d59f0f2ef3a8c977f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.985008 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28face87091b49d2971217618c95d2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.985805 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_76059d1e191546928659040fec3bdf30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.986426 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76059d1e191546928659040fec3bdf30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.987437 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:04.988013 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.990424 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_73193142c4d04b1f83e12f1ac807a71f" } modify_external_catalogs: true
I20251029 08:52:04.990594 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_73193142c4d04b1f83e12f1ac807a71f" } modify_external_catalogs: true
I20251029 08:52:04.987908 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_159d7048427049ffb8b14f34b6bb78fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:04.986024 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (DEFAULT_TABLE table=default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:04.992295 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_159d7048427049ffb8b14f34b6bb78fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:04.992726 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e67ddf5ceb148dfad8f7be9f26cff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.994489 758 tablet_service.cc:1505] Processing CreateTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (DEFAULT_TABLE table=default.Table_bb8cc7be97b8484f81dfddb6bb10e37a [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.994894 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ed0a270b794e469d2fdbd0ea1d96f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.994014 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (DEFAULT_TABLE table=default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:04.997327 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10baa0547180493f872a23614cf4c608. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:04.997545 1193 raft_consensus.cc:515] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:04.999145 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:04.999409 1176 tablet_replica.cc:333] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:04.999640 1176 raft_consensus.cc:2243] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:04.999825 1176 raft_consensus.cc:2272] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:04.987040 749 tablet_service.cc:1505] Processing CreateTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (DEFAULT_TABLE table=default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.000308 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.000837 1176 ts_tablet_manager.cc:1916] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.008186 1175 tablet_replica.cc:333] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:05.009117 905 tablet_service.cc:1505] Processing CreateTablet for tablet 7cbce5d595364b4bad36d59fb532c237 (DEFAULT_TABLE table=default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.010130 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 954c264e600a46f19db40dde5180787d with delete_type TABLET_DATA_DELETED (replaced by tablet 2e67ddf5ceb148dfad8f7be9f26cff64) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.010274 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.011210 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]). Expected version 2 got 0
I20251029 08:52:05.012629 1176 ts_tablet_manager.cc:1929] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.012821 1176 log.cc:1199] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/904523f7164849139cddb1100e95b824
I20251029 08:52:05.013305 1176 ts_tablet_manager.cc:1950] T 904523f7164849139cddb1100e95b824 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.014415 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 904523f7164849139cddb1100e95b824 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
I20251029 08:52:05.017846 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:05.017963 720 leader_election.cc:343] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 954c264e600a46f19db40dde5180787d
I20251029 08:52:05.019004 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (DEFAULT_TABLE table=default.Table_03feaca359af4f749d12bfa146dc1859 [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:05.019611 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 because the tablet was not found. No further retry: Not found: Tablet not found: 2e67ddf5ceb148dfad8f7be9f26cff64
W20251029 08:52:05.019912 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f already in progress: creating tablet
I20251029 08:52:05.020576 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (DEFAULT_TABLE table=default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.021438 1044 tablet_service.cc:1505] Processing CreateTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (DEFAULT_TABLE table=default.Table_bb8cc7be97b8484f81dfddb6bb10e37a [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.021394 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 (DEFAULT_TABLE table=default.Table_4b89cba8e4a642dabf87ae4f9a509895 [id=13d97eefb49f42cc94cb040ed86f4aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.021663 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.021544 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.021489 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94738d48801a4fb4a7ad1f5c4022afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.022207 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e67ddf5ceb148dfad8f7be9f26cff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.022226 861 leader_election.cc:304] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:05.022439 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ed0a270b794e469d2fdbd0ea1d96f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.022459 1175 raft_consensus.cc:2243] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.026011 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.009696 897 tablet_service.cc:1505] Processing CreateTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (DEFAULT_TABLE table=default.Table_bb8cc7be97b8484f81dfddb6bb10e37a [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.027909 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.028518 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ed0a270b794e469d2fdbd0ea1d96f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.029726 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "954c264e600a46f19db40dde5180787d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:05.032328 901 tablet_service.cc:1505] Processing CreateTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.032931 1035 tablet_service.cc:1505] Processing CreateTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb (DEFAULT_TABLE table=default.Table_ca96f0b4ad79470a8599b4e0e8cac23d [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.033711 756 tablet_service.cc:1505] Processing CreateTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb (DEFAULT_TABLE table=default.Table_ca96f0b4ad79470a8599b4e0e8cac23d [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.033811 1179 tablet_replica.cc:333] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.033805 903 tablet_service.cc:1505] Processing CreateTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb (DEFAULT_TABLE table=default.Table_ca96f0b4ad79470a8599b4e0e8cac23d [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.033982 1179 raft_consensus.cc:2243] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.034119 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add0605edf5c4511ae4c8d9cb7f0babb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.034188 1179 raft_consensus.cc:2272] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.035359 762 tablet_service.cc:1505] Processing CreateTablet for tablet be706771a95e45e5a20a15614eadfe1a (DEFAULT_TABLE table=default.Table_71c5f2a1b15b4d1e95c616783cd6b837 [id=e20293dc36d54acf9ee6f9bf7ac55834]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.035713 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be706771a95e45e5a20a15614eadfe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.036414 893 tablet_service.cc:1505] Processing CreateTablet for tablet be706771a95e45e5a20a15614eadfe1a (DEFAULT_TABLE table=default.Table_71c5f2a1b15b4d1e95c616783cd6b837 [id=e20293dc36d54acf9ee6f9bf7ac55834]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.037742 1175 raft_consensus.cc:2272] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.038618 1193 leader_election.cc:290] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.040297 1193 raft_consensus.cc:2804] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:05.041476 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2e15dbeda3ab4c4b8473ccd12597b270" } modify_external_catalogs: true
W20251029 08:52:05.041612 718 leader_election.cc:343] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.041671 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2e15dbeda3ab4c4b8473ccd12597b270" } modify_external_catalogs: true
I20251029 08:52:05.041766 718 leader_election.cc:304] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.042191 1323 raft_consensus.cc:2749] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.041827 1193 raft_consensus.cc:493] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:05.042857 1179 ts_tablet_manager.cc:1916] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.043111 1193 raft_consensus.cc:3060] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.044296 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 25b2e13184de4c25a5f14bc78d871bda with delete_type TABLET_DATA_DELETED (replaced by tablet 983a406aeed243f29ab322f94a0f5239) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.044463 1175 ts_tablet_manager.cc:1916] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.044507 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.044924 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.044930 1048 tablet_service.cc:1505] Processing CreateTablet for tablet be706771a95e45e5a20a15614eadfe1a (DEFAULT_TABLE table=default.Table_71c5f2a1b15b4d1e95c616783cd6b837 [id=e20293dc36d54acf9ee6f9bf7ac55834]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.045367 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be706771a95e45e5a20a15614eadfe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.052162 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 63dfcd944c1f487b8911ae383770d930 (DEFAULT_TABLE table=default.Table_e0316d58fa5c49d0a294cd9a37632924 [id=acafbaf717604c2e87a1b9e6263dcf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.052386 892 tablet_service.cc:1505] Processing CreateTablet for tablet 63dfcd944c1f487b8911ae383770d930 (DEFAULT_TABLE table=default.Table_e0316d58fa5c49d0a294cd9a37632924 [id=acafbaf717604c2e87a1b9e6263dcf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.057089 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc95257c937454aa6cd55b5a9394a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.057551 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add0605edf5c4511ae4c8d9cb7f0babb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.058071 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e67ddf5ceb148dfad8f7be9f26cff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.062608 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94738d48801a4fb4a7ad1f5c4022afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.062994 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be706771a95e45e5a20a15614eadfe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.063725 760 tablet_service.cc:1505] Processing CreateTablet for tablet 63dfcd944c1f487b8911ae383770d930 (DEFAULT_TABLE table=default.Table_e0316d58fa5c49d0a294cd9a37632924 [id=acafbaf717604c2e87a1b9e6263dcf48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.064839 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63dfcd944c1f487b8911ae383770d930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.066507 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63dfcd944c1f487b8911ae383770d930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.068634 1176 tablet_replica.cc:333] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.068979 1323 raft_consensus.cc:493] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:05.069303 1358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:05.075628 1323 raft_consensus.cc:515] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.076380 1323 leader_election.cc:290] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.077446 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25b2e13184de4c25a5f14bc78d871bda" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.080060 749 tablet_service.cc:1505] Processing CreateTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.081001 1178 tablet_bootstrap.cc:492] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.081198 1178 ts_tablet_manager.cc:1403] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.003s
I20251029 08:52:05.082072 1178 raft_consensus.cc:359] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.082368 1178 raft_consensus.cc:385] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.082499 1176 raft_consensus.cc:2243] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.082707 1178 raft_consensus.cc:740] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.083003 1178 consensus_queue.cc:260] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.084822 1178 ts_tablet_manager.cc:1434] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:05.085892 1178 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.086067 1176 raft_consensus.cc:2272] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.087162 1176 ts_tablet_manager.cc:1916] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.089186 1071 raft_consensus.cc:2468] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:05.089249 1179 ts_tablet_manager.cc:1929] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.089774 720 leader_election.cc:304] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:05.089774 1179 log.cc:1199] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/cc1d1d5b2a9e4051bbcd25be1578ee73
I20251029 08:52:05.090149 1323 raft_consensus.cc:2804] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:05.090242 1323 raft_consensus.cc:493] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:05.090339 1323 raft_consensus.cc:3060] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.091583 1179 ts_tablet_manager.cc:1950] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:05.093221 1323 raft_consensus.cc:515] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.094506 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25b2e13184de4c25a5f14bc78d871bda" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:05.094712 1071 raft_consensus.cc:3060] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.097839 1071 raft_consensus.cc:2468] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:05.098340 720 leader_election.cc:304] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:05.098664 1266 raft_consensus.cc:2804] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.101346 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (table default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]) successfully deleted
I20251029 08:52:05.101797 1323 leader_election.cc:290] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.103567 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63dfcd944c1f487b8911ae383770d930. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.103586 718 leader_election.cc:343] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:05.104131 1179 tablet_replica.cc:333] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:05.104910 718 leader_election.cc:343] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:05.107177 1179 raft_consensus.cc:2243] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.108112 1179 raft_consensus.cc:2272] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.109050 1179 ts_tablet_manager.cc:1916] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.114037 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94738d48801a4fb4a7ad1f5c4022afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.117826 1179 ts_tablet_manager.cc:1929] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.118201 1179 log.cc:1199] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:05.118597 1179 ts_tablet_manager.cc:1950] T b4ceeaba655542dab2556d3988b6e841 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:05.118983 1361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:05.119261 1323 raft_consensus.cc:493] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.119453 1323 raft_consensus.cc:515] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.119911 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b4ceeaba655542dab2556d3988b6e841 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:05.120309 1179 tablet_replica.cc:333] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.120580 1323 leader_election.cc:290] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.120741 1179 raft_consensus.cc:2243] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.120965 1179 raft_consensus.cc:2272] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.120945 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:05.121573 1179 ts_tablet_manager.cc:1916] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.122839 1178 tablet_bootstrap.cc:654] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.122828 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 13eeb2e8951b421e9d6f8f228a956e3d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:05.120921 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13eeb2e8951b421e9d6f8f228a956e3d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.123523 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.123751 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1de2f068a9d64b31b5211cd9c5791707"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:05.124090 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_69a6a8b8bfed4039a0833b799b013493" } new_table_name: "default.Table_753225d529ba4e54b0b04b6388919367" modify_external_catalogs: true
W20251029 08:52:05.124399 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de2f068a9d64b31b5211cd9c5791707 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.125581 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9869b4e9e33342c4bc5fe57a67b675ca" } new_table_name: "default.Table_ec785acfb6194e4c80654e0644b886a8" modify_external_catalogs: true
W20251029 08:52:05.123519 720 leader_election.cc:343] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.126057 756 tablet_service.cc:1552] Processing DeleteTablet for tablet be476c9994ed4dc28f606955b16bc4de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.126965 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.130117 1179 ts_tablet_manager.cc:1929] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.130295 1179 log.cc:1199] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:05.130749 1179 ts_tablet_manager.cc:1950] T 25b2e13184de4c25a5f14bc78d871bda P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:05.132149 1179 tablet_replica.cc:333] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.132416 1179 raft_consensus.cc:2243] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.132616 1179 raft_consensus.cc:2272] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.133379 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 25b2e13184de4c25a5f14bc78d871bda (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
I20251029 08:52:05.133560 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a58b4ac66df64b0db81210ce141ece47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:05.133940 1179 ts_tablet_manager.cc:1916] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.134363 1183 tablet_bootstrap.cc:492] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.134536 1183 ts_tablet_manager.cc:1403] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.228s user 0.008s sys 0.000s
W20251029 08:52:05.134848 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a58b4ac66df64b0db81210ce141ece47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.135362 1183 raft_consensus.cc:359] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.135577 1183 raft_consensus.cc:385] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.135645 1183 raft_consensus.cc:740] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.135885 1183 consensus_queue.cc:260] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.136865 1183 ts_tablet_manager.cc:1434] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:05.137317 1183 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.137714 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_effa05baab254c59bf4b358171091f33" } new_table_name: "default.Table_8c0fd828d9d14377aacc896765c045e3" modify_external_catalogs: true
I20251029 08:52:05.139503 1183 tablet_bootstrap.cc:654] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.144132 1179 ts_tablet_manager.cc:1929] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.144419 1179 log.cc:1199] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/06eb710809754e7abd00a345e8ae591a
W20251029 08:52:05.144603 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 50637.9us
I20251029 08:52:05.144963 1179 ts_tablet_manager.cc:1950] T 06eb710809754e7abd00a345e8ae591a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:05.146116 1179 tablet_replica.cc:333] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.146306 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 06eb710809754e7abd00a345e8ae591a (table default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]) successfully deleted
I20251029 08:52:05.146659 1179 raft_consensus.cc:2243] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.146909 1179 raft_consensus.cc:2272] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.147763 1179 ts_tablet_manager.cc:1916] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:05.147083 583 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b -> Peer f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435): Couldn't send request to peer f718bd0f00004f20a89d028a2fa8e6b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251029 08:52:05.151372 1176 ts_tablet_manager.cc:1929] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.151521 1176 log.cc:1199] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/954c264e600a46f19db40dde5180787d
I20251029 08:52:05.152638 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:05.153191 594 catalog_manager.cc:6719] ReplaceTablet: tablet 55842c18dbef4513a61dd9629f2e5a4c deleted and replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d
I20251029 08:52:05.154438 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bb6b86f39c994ea5930a6a970830093f
I20251029 08:52:05.155274 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a57acf768e244dea851580de4ce3b4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:05.155510 1179 ts_tablet_manager.cc:1929] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.155639 1179 log.cc:1199] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/6e2c32b84201470aa0dd697a5da12117
I20251029 08:52:05.155965 1179 ts_tablet_manager.cc:1950] T 6e2c32b84201470aa0dd697a5da12117 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:05.155958 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a57acf768e244dea851580de4ce3b4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.157647 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 6e2c32b84201470aa0dd697a5da12117 (table default.Table_eedf056f49564746a9c2c87482f5db42 [id=3a1661ecd47240a79cf75ebbcaf6acf2]) successfully deleted
I20251029 08:52:05.158658 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_271cf79d78b345b99c5f288178a0d0ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:05.160216 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.154459 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:05.160465 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.160984 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3d66a4f6f5af4c4688c532261fc9ccfa" } new_table_name: "default.Table_b3fef48b968b4888a66108a7e79d0ef7" modify_external_catalogs: true
I20251029 08:52:05.162459 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add0605edf5c4511ae4c8d9cb7f0babb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.162561 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_271cf79d78b345b99c5f288178a0d0ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:05.163833 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9165c4bf484a48b4a0b7246cb11571bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9165c4bf484a48b4a0b7246cb11571bf already in progress: creating tablet
I20251029 08:52:05.164877 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.166018 1183 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.166157 1183 ts_tablet_manager.cc:1403] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20251029 08:52:05.167090 1183 raft_consensus.cc:359] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.167299 1183 raft_consensus.cc:385] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.167392 1183 raft_consensus.cc:740] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.167649 1183 consensus_queue.cc:260] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:05.172191 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 55842c18dbef4513a61dd9629f2e5a4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55842c18dbef4513a61dd9629f2e5a4c already in progress: creating tablet
W20251029 08:52:05.174001 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bb6b86f39c994ea5930a6a970830093f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb6b86f39c994ea5930a6a970830093f already in progress: creating tablet
W20251029 08:52:05.173593 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 204438us
W20251029 08:52:05.175594 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 13eeb2e8951b421e9d6f8f228a956e3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13eeb2e8951b421e9d6f8f228a956e3d already in progress: creating tablet
I20251029 08:52:05.175580 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.176867 1183 ts_tablet_manager.cc:1434] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251029 08:52:05.177887 1183 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.181107 1183 tablet_bootstrap.cc:654] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.183539 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.183756 756 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.183995 1176 ts_tablet_manager.cc:1950] T 954c264e600a46f19db40dde5180787d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.186273 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbce5d595364b4bad36d59fb532c237. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.186630 1177 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.186795 1177 ts_tablet_manager.cc:1403] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.396s user 0.005s sys 0.011s
I20251029 08:52:05.187589 1353 raft_consensus.cc:2804] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.187759 1353 raft_consensus.cc:697] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:05.191008 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 954c264e600a46f19db40dde5180787d (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
I20251029 08:52:05.192574 1266 raft_consensus.cc:697] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:05.186210 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51bb01c123c44fb69b2f174fca5d7a13" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:05.194530 720 consensus_peers.cc:597] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:05.194712 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13eeb2e8951b421e9d6f8f228a956e3d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:05.197654 1342 raft_consensus.cc:493] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:05.197714 718 leader_election.cc:343] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.197815 1342 raft_consensus.cc:515] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.198637 1342 leader_election.cc:290] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:05.198992 900 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.199698 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.200490 1176 tablet_replica.cc:333] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.200719 1266 consensus_queue.cc:237] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.202979 718 leader_election.cc:304] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.203187 1178 tablet_bootstrap.cc:492] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.203341 1178 ts_tablet_manager.cc:1403] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.118s user 0.001s sys 0.009s
I20251029 08:52:05.203739 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e339c9ed26a48a2a9fe172201c5cc60" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:05.203909 784 raft_consensus.cc:2468] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:05.204200 1178 raft_consensus.cc:359] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:05.204351 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e339c9ed26a48a2a9fe172201c5cc60 consensus state has no leader
I20251029 08:52:05.204413 1178 raft_consensus.cc:385] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.204492 1178 raft_consensus.cc:740] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.204694 1178 consensus_queue.cc:260] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.204785 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb2ad1c3e1ce4bebab6c5d2df73cd8fb" } new_table_name: "default.Table_9ccaa6622c504f60b75e84efc44f6890" modify_external_catalogs: true
I20251029 08:52:05.206513 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1087b19fb3a74025b8269c7eb247bc14" } modify_external_catalogs: true
I20251029 08:52:05.206725 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1087b19fb3a74025b8269c7eb247bc14" } modify_external_catalogs: true
W20251029 08:52:05.207337 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74fe974a368a4afe8a5da6dfd1036400 has no consensus state
I20251029 08:52:05.207681 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_15261ce913ae430d8ea68e18a42df64a" } modify_external_catalogs: true
I20251029 08:52:05.207859 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_15261ce913ae430d8ea68e18a42df64a" } modify_external_catalogs: true
W20251029 08:52:05.208747 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdcc9a441c6a4d82a58262f4fdb3f00d has no consensus state
I20251029 08:52:05.210726 1323 raft_consensus.cc:2749] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.211515 1178 ts_tablet_manager.cc:1434] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:05.211916 1178 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.215790 1178 tablet_bootstrap.cc:654] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.217015 1183 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.218086 1183 ts_tablet_manager.cc:1403] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s
I20251029 08:52:05.218616 1176 raft_consensus.cc:2243] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:05.198989 1177 raft_consensus.cc:359] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.220461 1176 pending_rounds.cc:70] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:05.220608 1176 pending_rounds.cc:77] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037581626109952 op_type: NO_OP noop_request { }
I20251029 08:52:05.220809 1176 raft_consensus.cc:2889] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:05.220988 1176 raft_consensus.cc:2272] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.218379 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.218590 998 leader_election.cc:304] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:05.221662 1179 tablet_replica.cc:333] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.221930 1179 raft_consensus.cc:2243] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.222138 1179 raft_consensus.cc:2272] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.222065 1183 raft_consensus.cc:359] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.222306 1183 raft_consensus.cc:385] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.222383 1183 raft_consensus.cc:740] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.222419 1342 raft_consensus.cc:2804] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:05.222540 1342 raft_consensus.cc:493] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251029 08:52:05.222728 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8daa66608cf94913afb94ef661b3c898 has no consensus state
I20251029 08:52:05.222635 1183 consensus_queue.cc:260] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.222654 1342 raft_consensus.cc:3060] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:05.223548 1372 consensus_queue.cc:1175] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251029 08:52:05.227761 1176 ts_tablet_manager.cc:1916] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.229292 1179 ts_tablet_manager.cc:1916] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.230119 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e339c9ed26a48a2a9fe172201c5cc60" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:05.231981 593 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]). Expected version 2 got 0
I20251029 08:52:05.234491 1177 raft_consensus.cc:385] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:05.234683 861 leader_election.cc:343] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 51bb01c123c44fb69b2f174fca5d7a13
W20251029 08:52:05.234836 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 44605.3us
I20251029 08:52:05.235178 597 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:05.236377 1193 raft_consensus.cc:515] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.237514 1043 tablet_service.cc:1505] Processing CreateTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 (DEFAULT_TABLE table=default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.237895 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc307b14aff4c5e937a5151aab1dfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.238269 1176 ts_tablet_manager.cc:1929] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:05.238381 1176 log.cc:1199] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/25b2e13184de4c25a5f14bc78d871bda
I20251029 08:52:05.238814 1176 ts_tablet_manager.cc:1950] T 25b2e13184de4c25a5f14bc78d871bda P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.239550 1042 tablet_service.cc:1505] Processing CreateTablet for tablet ca5bfc2ee3c6491eb6857f6b9855daf9 (DEFAULT_TABLE table=default.Table_b58dc38f334840779d240a10ccb0d8db [id=01723803a27d472b90b05ba626327d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.239673 1176 tablet_replica.cc:333] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.239857 1176 raft_consensus.cc:2243] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.239961 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5bfc2ee3c6491eb6857f6b9855daf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.240324 1183 ts_tablet_manager.cc:1434] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251029 08:52:05.200217 1353 consensus_queue.cc:237] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.242172 1183 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.242669 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 25b2e13184de4c25a5f14bc78d871bda (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
I20251029 08:52:05.244047 597 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:05.244567 1183 tablet_bootstrap.cc:654] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.245780 1193 leader_election.cc:290] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:05.246999 861 leader_election.cc:343] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.253333 1177 raft_consensus.cc:740] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.253409 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.246799 1342 raft_consensus.cc:515] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.253633 861 leader_election.cc:304] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:05.253901 924 raft_consensus.cc:2468] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:05.254506 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14adc0f009d94287b92eda7b171c7c13" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:05.254653 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.254706 1071 raft_consensus.cc:3060] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.254782 1342 leader_election.cc:290] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:05.241967 749 tablet_service.cc:1505] Processing CreateTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 (DEFAULT_TABLE table=default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.255586 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc307b14aff4c5e937a5151aab1dfb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.256394 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dee931aa035748d08b1260e8ab3fca5c consensus state has no leader
I20251029 08:52:05.254577 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14adc0f009d94287b92eda7b171c7c13" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:05.260244 590 master_service.cc:954] ReplaceTablet: received request to replace tablet a61235da8c7a4f31b29d5d50c53c347d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:05.260291 784 raft_consensus.cc:3060] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.261605 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 79a3ef8389944c32bba7b3e598824118
I20251029 08:52:05.263013 784 raft_consensus.cc:2468] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:05.263207 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:05.255596 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e339c9ed26a48a2a9fe172201c5cc60" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:05.263792 783 raft_consensus.cc:3060] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.266394 758 tablet_service.cc:1505] Processing CreateTablet for tablet ca5bfc2ee3c6491eb6857f6b9855daf9 (DEFAULT_TABLE table=default.Table_b58dc38f334840779d240a10ccb0d8db [id=01723803a27d472b90b05ba626327d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.266556 783 raft_consensus.cc:2468] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:05.266860 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5bfc2ee3c6491eb6857f6b9855daf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.266824 756 tablet_service.cc:1505] Processing CreateTablet for tablet fa8884bb8e64474d936c8825d1ef3545 (DEFAULT_TABLE table=default.Table_dae092a6bac84104903dcf04163fc4d3 [id=064ab59757e94a9089fed2c7b3425476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.247166 1178 tablet_bootstrap.cc:492] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.267202 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8884bb8e64474d936c8825d1ef3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.267300 1178 ts_tablet_manager.cc:1403] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.001s
I20251029 08:52:05.267274 755 tablet_service.cc:1505] Processing CreateTablet for tablet 79a3ef8389944c32bba7b3e598824118 (DEFAULT_TABLE table=default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.267572 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.240015 1176 raft_consensus.cc:2272] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.268168 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a3ef8389944c32bba7b3e598824118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.268359 747 tablet_service.cc:1505] Processing CreateTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 (DEFAULT_TABLE table=default.Table_5288863701d147a28d9ee2a2d5e1754f [id=38e7e568f136480caaa0ee51ea6927f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.268915 998 leader_election.cc:304] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:05.269315 1041 tablet_service.cc:1505] Processing CreateTablet for tablet fa8884bb8e64474d936c8825d1ef3545 (DEFAULT_TABLE table=default.Table_dae092a6bac84104903dcf04163fc4d3 [id=064ab59757e94a9089fed2c7b3425476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.269418 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8626877e3cfc4cb48f6caa92638b4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.269702 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8884bb8e64474d936c8825d1ef3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.270390 1342 raft_consensus.cc:2804] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.270540 1342 raft_consensus.cc:697] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:05.270900 1342 consensus_queue.cc:237] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.273669 901 tablet_service.cc:1505] Processing CreateTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 (DEFAULT_TABLE table=default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.261644 1177 consensus_queue.cc:260] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.275342 1193 raft_consensus.cc:2749] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.276002 925 raft_consensus.cc:1275] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Refusing update from remote peer 7a8d80098fcb40ada903c3b2bacc2ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:05.276901 1360 raft_consensus.cc:493] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.279292 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc307b14aff4c5e937a5151aab1dfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.279330 1179 ts_tablet_manager.cc:1929] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.279815 891 tablet_service.cc:1505] Processing CreateTablet for tablet fa8884bb8e64474d936c8825d1ef3545 (DEFAULT_TABLE table=default.Table_dae092a6bac84104903dcf04163fc4d3 [id=064ab59757e94a9089fed2c7b3425476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.280180 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.280973 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.281080 1051 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (DEFAULT_TABLE table=default.Table_8c0fd828d9d14377aacc896765c045e3 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.281391 1035 tablet_service.cc:1505] Processing CreateTablet for tablet dee931aa035748d08b1260e8ab3fca5c (DEFAULT_TABLE table=default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.281721 1179 log.cc:1199] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:05.281900 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc9a441c6a4d82a58262f4fdb3f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.282524 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 8daa66608cf94913afb94ef661b3c898 (DEFAULT_TABLE table=default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.282835 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 79a3ef8389944c32bba7b3e598824118 (DEFAULT_TABLE table=default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.282914 1038 tablet_service.cc:1505] Processing CreateTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.285449 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 (DEFAULT_TABLE table=default.Table_5288863701d147a28d9ee2a2d5e1754f [id=38e7e568f136480caaa0ee51ea6927f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.286387 896 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.286803 752 tablet_service.cc:1505] Processing CreateTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.291270 760 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.286834 1360 raft_consensus.cc:515] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.294761 1175 ts_tablet_manager.cc:1929] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.294934 895 tablet_service.cc:1505] Processing CreateTablet for tablet ca5bfc2ee3c6491eb6857f6b9855daf9 (DEFAULT_TABLE table=default.Table_b58dc38f334840779d240a10ccb0d8db [id=01723803a27d472b90b05ba626327d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.295686 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.297703 861 leader_election.cc:304] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:05.298585 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea79bd4e6844b7abac623824cabbeeb" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:05.299377 1177 ts_tablet_manager.cc:1434] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.088s user 0.002s sys 0.000s
I20251029 08:52:05.300047 1175 log.cc:1199] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/954c264e600a46f19db40dde5180787d
I20251029 08:52:05.304064 1193 raft_consensus.cc:2804] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.305668 1177 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.305604 897 tablet_service.cc:1505] Processing CreateTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 (DEFAULT_TABLE table=default.Table_5288863701d147a28d9ee2a2d5e1754f [id=38e7e568f136480caaa0ee51ea6927f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.307348 749 tablet_service.cc:1505] Processing CreateTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 (DEFAULT_TABLE table=default.Table_ca7b76be24b04d34b006fef05f3df689 [id=9eab4fb7fff44c10854a84cbf994f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.307840 900 tablet_service.cc:1505] Processing CreateTablet for tablet 79a3ef8389944c32bba7b3e598824118 (DEFAULT_TABLE table=default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.307870 889 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (DEFAULT_TABLE table=default.Table_8c0fd828d9d14377aacc896765c045e3 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.308892 896 tablet_service.cc:1505] Processing CreateTablet for tablet 8daa66608cf94913afb94ef661b3c898 (DEFAULT_TABLE table=default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:05.309089 861 leader_election.cc:343] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.312224 894 tablet_service.cc:1505] Processing CreateTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 (DEFAULT_TABLE table=default.Table_ec785acfb6194e4c80654e0644b886a8 [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.312964 890 tablet_service.cc:1505] Processing CreateTablet for tablet dee931aa035748d08b1260e8ab3fca5c (DEFAULT_TABLE table=default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.314275 1360 leader_election.cc:290] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.314512 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5bfc2ee3c6491eb6857f6b9855daf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.316448 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8daa66608cf94913afb94ef661b3c898. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.316510 1193 raft_consensus.cc:697] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:05.316565 762 tablet_service.cc:1505] Processing CreateTablet for tablet dee931aa035748d08b1260e8ab3fca5c (DEFAULT_TABLE table=default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.316917 745 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (DEFAULT_TABLE table=default.Table_8c0fd828d9d14377aacc896765c045e3 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.317035 1193 consensus_queue.cc:237] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.317232 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8daa66608cf94913afb94ef661b3c898 (DEFAULT_TABLE table=default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.317576 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a1d5e1f68e4f0e8b78641bbdc98123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.318344 1175 ts_tablet_manager.cc:1950] T 954c264e600a46f19db40dde5180787d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:05.319034 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 954c264e600a46f19db40dde5180787d (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
W20251029 08:52:05.320123 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e67ddf5ceb148dfad8f7be9f26cff64 already in progress: creating tablet
I20251029 08:52:05.320554 746 tablet_service.cc:1505] Processing CreateTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 (DEFAULT_TABLE table=default.Table_ec785acfb6194e4c80654e0644b886a8 [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.320926 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fe974a368a4afe8a5da6dfd1036400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.321988 1178 raft_consensus.cc:359] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.322201 1178 raft_consensus.cc:385] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.322293 1178 raft_consensus.cc:740] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.322557 1178 consensus_queue.cc:260] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.323165 1178 ts_tablet_manager.cc:1434] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.056s user 0.001s sys 0.001s
I20251029 08:52:05.317056 888 tablet_service.cc:1505] Processing CreateTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.323503 1178 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.323639 1366 consensus_queue.cc:1048] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20251029 08:52:05.328374 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8daa66608cf94913afb94ef661b3c898. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.330778 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8884bb8e64474d936c8825d1ef3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.331245 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:05.332607 902 tablet_service.cc:1505] Processing CreateTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 (DEFAULT_TABLE table=default.Table_ca7b76be24b04d34b006fef05f3df689 [id=9eab4fb7fff44c10854a84cbf994f667]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:05.333909 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f already in progress: creating tablet
W20251029 08:52:05.334612 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7cbce5d595364b4bad36d59fb532c237 because the tablet was not found. No further retry: Not found: Tablet not found: 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:05.335618 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc9a441c6a4d82a58262f4fdb3f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.335790 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2d11b75de7d448c68fd63ab1ae81a464 (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.338374 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4ba094cbb1b04d25b294f69e9bac62be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.338913 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba094cbb1b04d25b294f69e9bac62be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.339324 1175 tablet_replica.cc:333] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:05.340266 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_63c241868b434e1d9af341e235b0554f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.340780 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c241868b434e1d9af341e235b0554f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.341621 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc9a441c6a4d82a58262f4fdb3f00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.342521 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee931aa035748d08b1260e8ab3fca5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.343259 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_91e03c5d65a24eda8ba54f91a6be6081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.344374 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91e03c5d65a24eda8ba54f91a6be6081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.345294 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757e34137e884d5ca5cd88fdd03244d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.345487 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.345978 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.347043 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.347503 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.347725 1176 ts_tablet_manager.cc:1916] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.349882 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0eecd4c045104f77bbeb969426549226" } modify_external_catalogs: true
I20251029 08:52:05.350039 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0eecd4c045104f77bbeb969426549226" } modify_external_catalogs: true
I20251029 08:52:05.350772 1178 tablet_bootstrap.cc:654] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.353787 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a3ef8389944c32bba7b3e598824118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.359253 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fe974a368a4afe8a5da6dfd1036400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.362025 1175 raft_consensus.cc:2243] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:05.365545 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8626877e3cfc4cb48f6caa92638b4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.366863 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 (DEFAULT_TABLE table=default.Table_ca7b76be24b04d34b006fef05f3df689 [id=9eab4fb7fff44c10854a84cbf994f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.367062 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea79bd4e6844b7abac623824cabbeeb" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.366888 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee931aa035748d08b1260e8ab3fca5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.368135 1177 tablet_bootstrap.cc:654] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.369108 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d11b75de7d448c68fd63ab1ae81a464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.369395 1069 raft_consensus.cc:2468] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:05.283085 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 (DEFAULT_TABLE table=default.Table_ec785acfb6194e4c80654e0644b886a8 [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.370371 861 leader_election.cc:304] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:05.370795 1353 raft_consensus.cc:2804] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:05.370914 1353 raft_consensus.cc:493] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:05.371002 1353 raft_consensus.cc:3060] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.372884 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2d11b75de7d448c68fd63ab1ae81a464 (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.375120 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_271cf79d78b345b99c5f288178a0d0ea" } modify_external_catalogs: true
I20251029 08:52:05.375365 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_271cf79d78b345b99c5f288178a0d0ea" } modify_external_catalogs: true
I20251029 08:52:05.375758 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d1dd83e945754532a1f9694d8c9d2e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:05.377118 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:05.377993 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a1d5e1f68e4f0e8b78641bbdc98123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.377113 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9ccaa6622c504f60b75e84efc44f6890" } modify_external_catalogs: true
I20251029 08:52:05.379424 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8626877e3cfc4cb48f6caa92638b4917. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.378855 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9ccaa6622c504f60b75e84efc44f6890" } modify_external_catalogs: true
I20251029 08:52:05.380868 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a3ef8389944c32bba7b3e598824118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.384181 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8daa66608cf94913afb94ef661b3c898. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.385352 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1dd83e945754532a1f9694d8c9d2e0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.386853 1071 raft_consensus.cc:2468] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:05.387336 1179 ts_tablet_manager.cc:1950] T 453911c5a29f43e7a21fa2c47ef264e8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:05.391018 1175 pending_rounds.cc:70] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:05.393886 1175 pending_rounds.cc:77] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037581790081024 op_type: NO_OP noop_request { }
I20251029 08:52:05.394094 1175 raft_consensus.cc:2889] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:05.394238 1175 raft_consensus.cc:2272] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.394320 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 453911c5a29f43e7a21fa2c47ef264e8 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
I20251029 08:52:05.399034 1175 ts_tablet_manager.cc:1916] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.400255 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757e34137e884d5ca5cd88fdd03244d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.400377 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a1d5e1f68e4f0e8b78641bbdc98123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.401532 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fe974a368a4afe8a5da6dfd1036400. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.406059 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a57b1dbbdf854bc6a6e1b5dc88b63bff already in progress: creating tablet
I20251029 08:52:05.416843 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee931aa035748d08b1260e8ab3fca5c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:05.417925 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 79a3ef8389944c32bba7b3e598824118 because the tablet was not found. No further retry: Not found: Tablet not found: 79a3ef8389944c32bba7b3e598824118
I20251029 08:52:05.420322 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 2d11b75de7d448c68fd63ab1ae81a464 (DEFAULT_TABLE table=default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.420936 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dae092a6bac84104903dcf04163fc4d3" } new_table_name: "default.Table_d3b4e5bde9434116b192d820a909c7b1" modify_external_catalogs: true
I20251029 08:52:05.421067 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d11b75de7d448c68fd63ab1ae81a464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.422448 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec785acfb6194e4c80654e0644b886a8" } new_table_name: "default.Table_611d65e05ca149d8910e6f16f99dce9c" modify_external_catalogs: true
I20251029 08:52:05.423660 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5288863701d147a28d9ee2a2d5e1754f" } new_table_name: "default.Table_c43c6468d7c447b28f338ce82a1df720" modify_external_catalogs: true
I20251029 08:52:05.426921 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet aee3c7f615c6491781373ced24adc5b0
I20251029 08:52:05.428041 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_aa83a84a696b4573b7ceeb7a274a698e" } new_table_name: "default.Table_cc9cefdf2fc24efb990767018ca474a8" modify_external_catalogs: true
I20251029 08:52:05.428246 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_54314bbc9f794709b664977bd9f32a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.428835 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54314bbc9f794709b664977bd9f32a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.430116 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_742ee97669df417aba70fdcea4983c94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.430656 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_742ee97669df417aba70fdcea4983c94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.432207 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 14adc0f009d94287b92eda7b171c7c13
I20251029 08:52:05.432955 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_77fe8da38eac41a6968a6e26fd81d040"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.433506 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77fe8da38eac41a6968a6e26fd81d040 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.435418 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b8818df4a5ac46bcb3921874a943a18a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.435916 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8818df4a5ac46bcb3921874a943a18a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.437158 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 14adc0f009d94287b92eda7b171c7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.438416 1179 tablet_replica.cc:333] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:05.438694 1179 raft_consensus.cc:2243] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.439347 1176 ts_tablet_manager.cc:1929] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.439519 1176 log.cc:1199] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/be476c9994ed4dc28f606955b16bc4de
I20251029 08:52:05.439504 1179 raft_consensus.cc:2272] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.440129 1179 ts_tablet_manager.cc:1916] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.446920 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet dee931aa035748d08b1260e8ab3fca5c
I20251029 08:52:05.440032 1176 ts_tablet_manager.cc:1950] T be476c9994ed4dc28f606955b16bc4de P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.455013 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet be476c9994ed4dc28f606955b16bc4de (table default.Table_1c46023036d34ee7be5bb30a14caa0e2 [id=6c75c2c51a93469489dfd59bce7906c6]) successfully deleted
I20251029 08:52:05.440162 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 14adc0f009d94287b92eda7b171c7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.450007 752 tablet_service.cc:1505] Processing CreateTablet for tablet 3295bb0b47714080a5c3f34049acd487 (DEFAULT_TABLE table=default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.456490 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3295bb0b47714080a5c3f34049acd487. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.457594 893 tablet_service.cc:1505] Processing CreateTablet for tablet 3295bb0b47714080a5c3f34049acd487 (DEFAULT_TABLE table=default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.459909 1366 raft_consensus.cc:493] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.459918 905 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.460093 1366 raft_consensus.cc:515] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.460613 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757e34137e884d5ca5cd88fdd03244d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.460922 1366 leader_election.cc:290] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.464814 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ca96f0b4ad79470a8599b4e0e8cac23d" } new_table_name: "default.Table_c1103092b7ae447dbfd1b7782cde6faa" modify_external_catalogs: true
I20251029 08:52:05.466391 747 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.453037 1178 tablet_bootstrap.cc:492] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.467700 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3295bb0b47714080a5c3f34049acd487. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.468465 1178 ts_tablet_manager.cc:1403] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.005s
I20251029 08:52:05.469065 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16a2b37e45e444caa3f5117ae9a25304" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:05.469698 718 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.451571 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 14adc0f009d94287b92eda7b171c7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.470013 1178 raft_consensus.cc:359] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.470213 1178 raft_consensus.cc:385] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.470853 1178 raft_consensus.cc:740] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.471225 1178 consensus_queue.cc:260] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.471557 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 3295bb0b47714080a5c3f34049acd487 (DEFAULT_TABLE table=default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.472035 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3295bb0b47714080a5c3f34049acd487. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.474751 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d11b75de7d448c68fd63ab1ae81a464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.475719 1178 ts_tablet_manager.cc:1434] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 08:52:05.478931 1178 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.480154 1353 raft_consensus.cc:515] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:05.481011 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9165c4bf484a48b4a0b7246cb11571bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9165c4bf484a48b4a0b7246cb11571bf already in progress: creating tablet
W20251029 08:52:05.482327 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 55842c18dbef4513a61dd9629f2e5a4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55842c18dbef4513a61dd9629f2e5a4c already in progress: creating tablet
I20251029 08:52:05.483100 1176 tablet_replica.cc:333] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.483325 1176 raft_consensus.cc:2243] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.483552 1176 raft_consensus.cc:2272] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.484167 1176 ts_tablet_manager.cc:1916] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.490948 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ba094cbb1b04d25b294f69e9bac62be" } modify_external_catalogs: true
I20251029 08:52:05.491186 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ba094cbb1b04d25b294f69e9bac62be" } modify_external_catalogs: true
W20251029 08:52:05.491880 1380 log.cc:927] Time spent T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20251029 08:52:05.495182 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.495756 1262 server_base.cc:1047] running on GCE node
I20251029 08:52:05.502198 1178 tablet_bootstrap.cc:654] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:05.496438 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:05.507930 1262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 08:52:05.508138 1353 leader_election.cc:290] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.508176 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea79bd4e6844b7abac623824cabbeeb" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:05.511461 783 raft_consensus.cc:3060] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.514436 783 raft_consensus.cc:2468] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:05.514763 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa8884bb8e64474d936c8825d1ef3545 consensus state has no leader
I20251029 08:52:05.515005 1366 raft_consensus.cc:493] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.514957 861 leader_election.cc:304] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:05.515134 1366 raft_consensus.cc:515] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.515785 1366 leader_election.cc:290] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.517023 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e2c32b84201470aa0dd697a5da12117" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:05.517612 718 leader_election.cc:343] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: Tablet not RUNNING: STOPPED
I20251029 08:52:05.520692 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a09dfe0f96c64010a72526b0f6cb3920" } modify_external_catalogs: true
I20251029 08:52:05.520915 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a09dfe0f96c64010a72526b0f6cb3920" } modify_external_catalogs: true
I20251029 08:52:05.522161 1176 ts_tablet_manager.cc:1929] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.522310 1176 log.cc:1199] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:05.522789 1176 ts_tablet_manager.cc:1950] T 13eeb2e8951b421e9d6f8f228a956e3d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.523869 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 13eeb2e8951b421e9d6f8f228a956e3d (table default.Table_2e15dbeda3ab4c4b8473ccd12597b270 [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:05.532222 1353 raft_consensus.cc:2804] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.532497 1353 raft_consensus.cc:697] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:05.532897 1353 consensus_queue.cc:237] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:05.533524 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74fe974a368a4afe8a5da6dfd1036400 consensus state has no leader
W20251029 08:52:05.540956 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bb6b86f39c994ea5930a6a970830093f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb6b86f39c994ea5930a6a970830093f already in progress: creating tablet
W20251029 08:52:05.543129 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 79a3ef8389944c32bba7b3e598824118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a3ef8389944c32bba7b3e598824118 already in progress: creating tablet
I20251029 08:52:05.543648 1175 ts_tablet_manager.cc:1929] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 08:52:05.544981 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a57b1dbbdf854bc6a6e1b5dc88b63bff already in progress: creating tablet
I20251029 08:52:05.546566 1176 tablet_replica.cc:333] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:05.546705 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8626877e3cfc4cb48f6caa92638b4917 consensus state has no leader
I20251029 08:52:05.548126 1176 raft_consensus.cc:2243] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.549132 1176 raft_consensus.cc:2272] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.558131 1176 ts_tablet_manager.cc:1916] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:05.560142 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73a891fb6034f989d29a3f837303d84 consensus state has no leader
I20251029 08:52:05.565531 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_390ea1069fb74e8fa400c7c309ec231e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.566370 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_390ea1069fb74e8fa400c7c309ec231e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.568301 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d71bb6e219494f90897bd2f8844f599e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.568830 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d71bb6e219494f90897bd2f8844f599e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.561815 1175 log.cc:1199] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/6e2c32b84201470aa0dd697a5da12117
I20251029 08:52:05.573086 1178 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.573276 1178 ts_tablet_manager.cc:1403] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.010s
I20251029 08:52:05.575321 1175 ts_tablet_manager.cc:1950] T 6e2c32b84201470aa0dd697a5da12117 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:05.576318 1175 tablet_replica.cc:333] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:05.576439 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 6e2c32b84201470aa0dd697a5da12117 (table default.Table_eedf056f49564746a9c2c87482f5db42 [id=3a1661ecd47240a79cf75ebbcaf6acf2]) successfully deleted
I20251029 08:52:05.576520 1175 raft_consensus.cc:2243] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.576678 1175 raft_consensus.cc:2272] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.576792 1178 raft_consensus.cc:359] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.577054 1178 raft_consensus.cc:385] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.577185 1178 raft_consensus.cc:740] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.577669 1178 consensus_queue.cc:260] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.579697 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16a2b37e45e444caa3f5117ae9a25304" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.584255 1178 ts_tablet_manager.cc:1434] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
W20251029 08:52:05.585525 720 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.585675 720 leader_election.cc:304] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.586067 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea79bd4e6844b7abac623824cabbeeb" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:05.586138 720 leader_election.cc:343] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 6e2c32b84201470aa0dd697a5da12117
I20251029 08:52:05.586248 720 leader_election.cc:304] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.586617 1266 raft_consensus.cc:2749] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.586812 1266 raft_consensus.cc:2749] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.589358 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 1 got 0
I20251029 08:52:05.589696 1175 ts_tablet_manager.cc:1916] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.590310 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d7165655d9634a7bb270da6738b6c0c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.590970 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7165655d9634a7bb270da6738b6c0c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.594985 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_685d1d31eb774418a8e997595f7e8ff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.595688 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685d1d31eb774418a8e997595f7e8ff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.597419 1176 ts_tablet_manager.cc:1929] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.598302 1176 log.cc:1199] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/14adc0f009d94287b92eda7b171c7c13
I20251029 08:52:05.598635 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e339c9ed26a48a2a9fe172201c5cc60" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:05.598846 924 raft_consensus.cc:3060] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.598925 1176 ts_tablet_manager.cc:1950] T 14adc0f009d94287b92eda7b171c7c13 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.600037 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]) successfully deleted
I20251029 08:52:05.597558 1178 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.601109 1262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:52:05.602896 1178 tablet_bootstrap.cc:654] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:05.607681 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet dee931aa035748d08b1260e8ab3fca5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee931aa035748d08b1260e8ab3fca5c already in progress: creating tablet
I20251029 08:52:05.608162 1176 tablet_replica.cc:333] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.608402 1176 raft_consensus.cc:2243] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.611824 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 1 got 0
I20251029 08:52:05.614111 1176 raft_consensus.cc:2272] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.615235 1176 ts_tablet_manager.cc:1916] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.623090 1176 ts_tablet_manager.cc:1929] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.626873 1262 hybrid_clock.cc:648] HybridClock initialized: now 1761727925626702 us; error 125 us; skew 500 ppm
I20251029 08:52:05.627627 1262 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:52:05.630244 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f6ce198ce16b46249069916455fe0165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.631191 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6ce198ce16b46249069916455fe0165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.631323 1183 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.631567 1183 ts_tablet_manager.cc:1403] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.389s user 0.025s sys 0.006s
I20251029 08:52:05.632195 1176 log.cc:1199] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/6e2c32b84201470aa0dd697a5da12117
I20251029 08:52:05.632627 1176 ts_tablet_manager.cc:1950] T 6e2c32b84201470aa0dd697a5da12117 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.632695 1183 raft_consensus.cc:359] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.632956 1183 raft_consensus.cc:385] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.633080 1183 raft_consensus.cc:740] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.633378 1183 consensus_queue.cc:260] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.634063 1183 ts_tablet_manager.cc:1434] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:05.634536 1183 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.636240 1069 raft_consensus.cc:3060] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.637106 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 6e2c32b84201470aa0dd697a5da12117 (table default.Table_eedf056f49564746a9c2c87482f5db42 [id=3a1661ecd47240a79cf75ebbcaf6acf2]) successfully deleted
I20251029 08:52:05.640758 1069 raft_consensus.cc:2468] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:05.641670 1183 tablet_bootstrap.cc:654] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.643365 1179 ts_tablet_manager.cc:1929] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.643590 1179 log.cc:1199] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/14adc0f009d94287b92eda7b171c7c13
I20251029 08:52:05.644057 1179 ts_tablet_manager.cc:1950] T 14adc0f009d94287b92eda7b171c7c13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:05.644188 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e2c32b84201470aa0dd697a5da12117 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.644985 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:05.646369 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]) successfully deleted
W20251029 08:52:05.646679 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet dee931aa035748d08b1260e8ab3fca5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee931aa035748d08b1260e8ab3fca5c already in progress: creating tablet
I20251029 08:52:05.647230 1177 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.648075 1183 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.648607 1183 ts_tablet_manager.cc:1403] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251029 08:52:05.649559 1183 raft_consensus.cc:359] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.649863 1183 raft_consensus.cc:385] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.649991 1183 raft_consensus.cc:740] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.650315 1183 consensus_queue.cc:260] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.651700 924 raft_consensus.cc:2468] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:05.652803 1183 ts_tablet_manager.cc:1434] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:05.655128 1175 ts_tablet_manager.cc:1929] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.655305 1175 log.cc:1199] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/51bb01c123c44fb69b2f174fca5d7a13
I20251029 08:52:05.655742 1175 ts_tablet_manager.cc:1950] T 51bb01c123c44fb69b2f174fca5d7a13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:05.655508 1183 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.658092 1183 tablet_bootstrap.cc:654] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.658877 1177 ts_tablet_manager.cc:1403] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.359s user 0.057s sys 0.061s
I20251029 08:52:05.659281 1175 tablet_replica.cc:333] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:05.659796 1175 raft_consensus.cc:2243] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:05.660549 1175 pending_rounds.cc:70] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:05.660696 1175 pending_rounds.cc:77] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037582132150272 op_type: NO_OP noop_request { }
I20251029 08:52:05.661072 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 51bb01c123c44fb69b2f174fca5d7a13 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
W20251029 08:52:05.661504 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bb6b86f39c994ea5930a6a970830093f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb6b86f39c994ea5930a6a970830093f already in progress: creating tablet
W20251029 08:52:05.661857 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 55842c18dbef4513a61dd9629f2e5a4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55842c18dbef4513a61dd9629f2e5a4c already in progress: creating tablet
W20251029 08:52:05.662158 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 13eeb2e8951b421e9d6f8f228a956e3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13eeb2e8951b421e9d6f8f228a956e3d already in progress: creating tablet
W20251029 08:52:05.662482 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9165c4bf484a48b4a0b7246cb11571bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9165c4bf484a48b4a0b7246cb11571bf already in progress: creating tablet
W20251029 08:52:05.663020 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a57b1dbbdf854bc6a6e1b5dc88b63bff already in progress: creating tablet
W20251029 08:52:05.663352 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 79a3ef8389944c32bba7b3e598824118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a3ef8389944c32bba7b3e598824118 already in progress: creating tablet
I20251029 08:52:05.663568 1178 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
W20251029 08:52:05.663681 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet dee931aa035748d08b1260e8ab3fca5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee931aa035748d08b1260e8ab3fca5c already in progress: creating tablet
I20251029 08:52:05.663803 1178 ts_tablet_manager.cc:1403] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.006s
I20251029 08:52:05.664721 1175 raft_consensus.cc:2889] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:05.664703 1178 raft_consensus.cc:359] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.664877 1175 raft_consensus.cc:2272] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.659873 1177 raft_consensus.cc:359] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.665027 1178 raft_consensus.cc:385] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.665131 1177 raft_consensus.cc:385] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.665230 1177 raft_consensus.cc:740] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.665541 1177 consensus_queue.cc:260] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.665840 1178 raft_consensus.cc:740] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.666700 1177 ts_tablet_manager.cc:1434] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:05.666215 1178 consensus_queue.cc:260] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.667483 1177 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.669998 1177 tablet_bootstrap.cc:654] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.672186 1175 ts_tablet_manager.cc:1916] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.677071 1177 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.677270 1177 ts_tablet_manager.cc:1403] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20251029 08:52:05.678184 1177 raft_consensus.cc:359] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.678402 1177 raft_consensus.cc:385] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.678524 1177 raft_consensus.cc:740] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.679636 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c43c6468d7c447b28f338ce82a1df720" } modify_external_catalogs: true
I20251029 08:52:05.679675 1177 consensus_queue.cc:260] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.679872 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c43c6468d7c447b28f338ce82a1df720" } modify_external_catalogs: true
I20251029 08:52:05.680749 1177 ts_tablet_manager.cc:1434] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:05.681043 1177 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.687862 1183 tablet_bootstrap.cc:492] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.688683 1183 ts_tablet_manager.cc:1403] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.007s
I20251029 08:52:05.689780 1183 raft_consensus.cc:359] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.690096 1183 raft_consensus.cc:385] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.690248 1183 raft_consensus.cc:740] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.690609 1183 consensus_queue.cc:260] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.692457 1183 ts_tablet_manager.cc:1434] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:05.692937 1183 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.695834 1183 tablet_bootstrap.cc:654] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.696923 1178 ts_tablet_manager.cc:1434] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20251029 08:52:05.697757 1178 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.699566 1183 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
W20251029 08:52:05.701041 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add0605edf5c4511ae4c8d9cb7f0babb consensus state has no leader
I20251029 08:52:05.702335 1183 ts_tablet_manager.cc:1403] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251029 08:52:05.704159 591 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c4f081294ba549ac86b5ba5e3e487090
I20251029 08:52:05.704666 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_32228c67a14f44429dd7a7bbed1e1e34" } modify_external_catalogs: true
I20251029 08:52:05.704897 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_32228c67a14f44429dd7a7bbed1e1e34" } modify_external_catalogs: true
I20251029 08:52:05.705574 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 983a406aeed243f29ab322f94a0f5239
I20251029 08:52:05.706410 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6899fb76bad448cd9d1ca4299c5f7562" } modify_external_catalogs: true
I20251029 08:52:05.706610 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6899fb76bad448cd9d1ca4299c5f7562" } modify_external_catalogs: true
I20251029 08:52:05.707938 1178 tablet_bootstrap.cc:654] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.708446 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:05.709379 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 983a406aeed243f29ab322f94a0f5239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 983a406aeed243f29ab322f94a0f5239 already in progress: creating tablet
I20251029 08:52:05.709978 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:05.710835 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 983a406aeed243f29ab322f94a0f5239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 983a406aeed243f29ab322f94a0f5239 already in progress: creating tablet
I20251029 08:52:05.711102 590 master_service.cc:954] ReplaceTablet: received request to replace tablet b273c488f33c498890f1739380b18239 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:05.711261 1353 raft_consensus.cc:493] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.712035 752 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:05.712286 1176 tablet_replica.cc:333] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:05.712517 1176 raft_consensus.cc:2243] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:05.712694 1176 raft_consensus.cc:2272] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:05.713752 1176 ts_tablet_manager.cc:1916] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:05.720374 1353 raft_consensus.cc:515] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.721150 1353 leader_election.cc:290] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:05.721599 1183 raft_consensus.cc:359] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.721705 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:05.721782 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.721853 1183 raft_consensus.cc:385] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.724782 1183 raft_consensus.cc:740] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.725373 1183 consensus_queue.cc:260] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.726727 1176 ts_tablet_manager.cc:1929] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:05.726949 1176 log.cc:1199] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bb6b86f39c994ea5930a6a970830093f
I20251029 08:52:05.727460 1176 ts_tablet_manager.cc:1950] T bb6b86f39c994ea5930a6a970830093f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:05.729157 1183 ts_tablet_manager.cc:1434] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251029 08:52:05.733765 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7e3d3e525fa34d06bb82e54fc6cc95af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:05.734504 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e3d3e525fa34d06bb82e54fc6cc95af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:05.734948 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bb6b86f39c994ea5930a6a970830093f (table default.Table_73193142c4d04b1f83e12f1ac807a71f [id=2f82075a67804ac48d6817dd2924b9a8]) successfully deleted
W20251029 08:52:05.735327 861 leader_election.cc:343] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:05.735630 861 leader_election.cc:343] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.739807 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cb1c7e6ae28b4dd48a2af79080df8e1b" } new_table_name: "default.Table_b9b1c4a272214856ab50337015cdbd7b" modify_external_catalogs: true
I20251029 08:52:05.741650 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 7e339c9ed26a48a2a9fe172201c5cc60 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:05.742015 861 leader_election.cc:304] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:05.742570 1177 tablet_bootstrap.cc:654] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.744346 1183 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:05.743078 1395 tablet.cc:1722] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:05.748484 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:05.749591 1262 webserver.cc:492] Webserver started at http://127.0.104.62:37745/ using document root <none> and password file <none>
I20251029 08:52:05.749827 1353 raft_consensus.cc:2749] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.751080 1262 fs_manager.cc:362] Metadata directory not provided
I20251029 08:52:05.751425 1262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:52:05.752456 1175 ts_tablet_manager.cc:1929] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:05.752890 1175 log.cc:1199] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/14adc0f009d94287b92eda7b171c7c13
I20251029 08:52:05.754050 1175 ts_tablet_manager.cc:1950] T 14adc0f009d94287b92eda7b171c7c13 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:05.755221 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 14adc0f009d94287b92eda7b171c7c13 (table default.Table_0eecd4c045104f77bbeb969426549226 [id=9a65ac9ea0f448a38b2098fcefd707dc]) successfully deleted
W20251029 08:52:05.755560 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6e2c32b84201470aa0dd697a5da12117 because the tablet was not found. No further retry: Not found: Tablet not found: 6e2c32b84201470aa0dd697a5da12117
W20251029 08:52:05.755818 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 983a406aeed243f29ab322f94a0f5239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 983a406aeed243f29ab322f94a0f5239 already in progress: creating tablet
I20251029 08:52:05.761152 1178 tablet_bootstrap.cc:492] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.770866 1178 ts_tablet_manager.cc:1403] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20251029 08:52:05.772235 1178 raft_consensus.cc:359] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.774000 1178 raft_consensus.cc:385] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.774310 1178 raft_consensus.cc:740] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.775641 1178 consensus_queue.cc:260] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.777176 1177 tablet_bootstrap.cc:492] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.777488 1177 ts_tablet_manager.cc:1403] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.004s
I20251029 08:52:05.778710 1177 raft_consensus.cc:359] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.778929 1177 raft_consensus.cc:385] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.779017 1177 raft_consensus.cc:740] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.779284 1177 consensus_queue.cc:260] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.779930 1177 ts_tablet_manager.cc:1434] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:05.780298 1177 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.780857 589 catalog_manager.cc:5649] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:05.781401 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 7e339c9ed26a48a2a9fe172201c5cc60 (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]). Expected version 1 got 0
I20251029 08:52:05.781534 1178 ts_tablet_manager.cc:1434] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 08:52:05.781867 1178 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.782214 1177 tablet_bootstrap.cc:654] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.783711 1178 tablet_bootstrap.cc:654] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.784852 1398 tablet.cc:1722] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:05.787257 752 tablet_service.cc:1505] Processing CreateTablet for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (DEFAULT_TABLE table=default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c [id=5dc0a7f470cb432dbd918d84b96dd58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.787782 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc57cd5a8d3c42ba90f2011084e8b4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.788321 892 tablet_service.cc:1505] Processing CreateTablet for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (DEFAULT_TABLE table=default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c [id=5dc0a7f470cb432dbd918d84b96dd58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.788668 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc57cd5a8d3c42ba90f2011084e8b4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.791326 890 tablet_service.cc:1505] Processing CreateTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.791679 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c5008d9df9419bb4b6d678c6dc27a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.792716 747 tablet_service.cc:1505] Processing CreateTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.793123 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c5008d9df9419bb4b6d678c6dc27a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.794441 902 tablet_service.cc:1505] Processing CreateTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 (DEFAULT_TABLE table=default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.794783 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc96ef1a94844a32923087c44a5bcfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.796525 749 tablet_service.cc:1505] Processing CreateTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 (DEFAULT_TABLE table=default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.796983 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc96ef1a94844a32923087c44a5bcfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.798368 895 tablet_service.cc:1505] Processing CreateTablet for tablet ee8117e04f5f40e98ea755dc461067dd (DEFAULT_TABLE table=default.Table_a57acf768e244dea851580de4ce3b4d3 [id=5730455f82a94d8698ac62979d87d9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.799386 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8117e04f5f40e98ea755dc461067dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.800355 748 tablet_service.cc:1505] Processing CreateTablet for tablet ee8117e04f5f40e98ea755dc461067dd (DEFAULT_TABLE table=default.Table_a57acf768e244dea851580de4ce3b4d3 [id=5730455f82a94d8698ac62979d87d9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.800755 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8117e04f5f40e98ea755dc461067dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.810245 745 tablet_service.cc:1505] Processing CreateTablet for tablet 215cd666c71a4179a19e298868a88e96 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.810774 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cd666c71a4179a19e298868a88e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.812211 892 tablet_service.cc:1505] Processing CreateTablet for tablet 215cd666c71a4179a19e298868a88e96 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.813431 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cd666c71a4179a19e298868a88e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.815728 1040 tablet_service.cc:1505] Processing CreateTablet for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (DEFAULT_TABLE table=default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c [id=5dc0a7f470cb432dbd918d84b96dd58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.817957 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.818923 1043 tablet_service.cc:1505] Processing CreateTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 (DEFAULT_TABLE table=default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.820641 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc57cd5a8d3c42ba90f2011084e8b4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.826180 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc96ef1a94844a32923087c44a5bcfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.832444 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c5008d9df9419bb4b6d678c6dc27a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.835392 1177 tablet_bootstrap.cc:492] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.835579 1177 ts_tablet_manager.cc:1403] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.001s
I20251029 08:52:05.835733 1178 tablet_bootstrap.cc:492] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.835981 1178 ts_tablet_manager.cc:1403] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.018s
I20251029 08:52:05.836820 1177 raft_consensus.cc:359] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.837061 1177 raft_consensus.cc:385] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.837184 1177 raft_consensus.cc:740] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.836966 1178 raft_consensus.cc:359] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.837384 1178 raft_consensus.cc:385] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.837513 1178 raft_consensus.cc:740] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.837474 1177 consensus_queue.cc:260] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.838614 1177 ts_tablet_manager.cc:1434] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251029 08:52:05.837894 1178 consensus_queue.cc:260] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.838929 1177 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.839594 1178 ts_tablet_manager.cc:1434] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:05.840642 1178 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.841174 1038 tablet_service.cc:1505] Processing CreateTablet for tablet ee8117e04f5f40e98ea755dc461067dd (DEFAULT_TABLE table=default.Table_a57acf768e244dea851580de4ce3b4d3 [id=5730455f82a94d8698ac62979d87d9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.841672 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee8117e04f5f40e98ea755dc461067dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.842422 1177 tablet_bootstrap.cc:654] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.842674 1178 tablet_bootstrap.cc:654] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.843549 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 (DEFAULT_TABLE table=default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.844290 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff4e625abf14fbd94a8102dfa5fc993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.845680 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 215cd666c71a4179a19e298868a88e96 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.846100 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cd666c71a4179a19e298868a88e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.851246 1178 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.851488 1178 ts_tablet_manager.cc:1403] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20251029 08:52:05.852458 1178 raft_consensus.cc:359] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.852731 1178 raft_consensus.cc:385] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.852867 1178 raft_consensus.cc:740] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.853224 1178 consensus_queue.cc:260] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.853938 1178 ts_tablet_manager.cc:1434] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:05.854293 1178 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.855255 1366 raft_consensus.cc:493] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.855417 1366 raft_consensus.cc:515] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.856003 1366 leader_election.cc:290] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.856789 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:05.856812 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:05.857306 718 leader_election.cc:343] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:05.857556 720 leader_election.cc:343] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.857654 720 leader_election.cc:304] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.858220 1366 raft_consensus.cc:2749] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.860652 1177 tablet_bootstrap.cc:492] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.860846 1177 ts_tablet_manager.cc:1403] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20251029 08:52:05.860997 1178 tablet_bootstrap.cc:654] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.861757 1177 raft_consensus.cc:359] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.862011 1177 raft_consensus.cc:385] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.862098 1177 raft_consensus.cc:740] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.862409 1177 consensus_queue.cc:260] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.862972 1177 ts_tablet_manager.cc:1434] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251029 08:52:05.863415 1177 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:05.864215 582 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet a73a891fb6034f989d29a3f837303d84 consensus state has no leader
I20251029 08:52:05.866147 1177 tablet_bootstrap.cc:654] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.872531 1178 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.872871 1178 ts_tablet_manager.cc:1403] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20251029 08:52:05.874158 1178 raft_consensus.cc:359] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.874439 1178 raft_consensus.cc:385] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.874545 1178 raft_consensus.cc:740] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.874859 1178 consensus_queue.cc:260] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.876001 1178 ts_tablet_manager.cc:1434] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:05.877112 1178 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.879086 1178 tablet_bootstrap.cc:654] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.884936 1177 tablet_bootstrap.cc:492] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.885131 1177 ts_tablet_manager.cc:1403] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20251029 08:52:05.886312 1177 raft_consensus.cc:359] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.886550 1177 raft_consensus.cc:385] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.886642 1177 raft_consensus.cc:740] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.886953 1177 consensus_queue.cc:260] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.887670 1177 ts_tablet_manager.cc:1434] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:05.888069 1178 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.888289 1178 ts_tablet_manager.cc:1403] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20251029 08:52:05.888870 1177 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.889268 1178 raft_consensus.cc:359] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.889549 1178 raft_consensus.cc:385] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.889701 1178 raft_consensus.cc:740] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.890033 1178 consensus_queue.cc:260] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.890266 1366 raft_consensus.cc:493] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.890440 1366 raft_consensus.cc:515] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.890812 1178 ts_tablet_manager.cc:1434] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:05.891525 1177 tablet_bootstrap.cc:654] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.891691 1366 leader_election.cc:290] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:05.891721 1178 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.892562 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0b459b366cb402db2e7e8fee2b3e855" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.892644 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0b459b366cb402db2e7e8fee2b3e855" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:05.893229 720 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:05.893371 718 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:05.893496 718 leader_election.cc:304] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:05.893944 1366 raft_consensus.cc:2749] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.895385 1178 tablet_bootstrap.cc:654] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.899297 1177 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.899650 1177 ts_tablet_manager.cc:1403] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251029 08:52:05.900698 1177 raft_consensus.cc:359] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.900918 1177 raft_consensus.cc:385] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.901015 1178 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.901001 1177 raft_consensus.cc:740] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.901252 1178 ts_tablet_manager.cc:1403] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20251029 08:52:05.901289 1177 consensus_queue.cc:260] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.902218 1178 raft_consensus.cc:359] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.902488 1178 raft_consensus.cc:385] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.902609 1178 raft_consensus.cc:740] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.902987 1178 consensus_queue.cc:260] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.903275 1177 ts_tablet_manager.cc:1434] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:05.904242 1178 ts_tablet_manager.cc:1434] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:05.905205 1178 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.905639 1177 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.907182 1177 tablet_bootstrap.cc:654] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.907219 1178 tablet_bootstrap.cc:654] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.908373 1183 tablet_bootstrap.cc:654] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.911720 752 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 (DEFAULT_TABLE table=default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.912343 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff4e625abf14fbd94a8102dfa5fc993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.916914 892 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 (DEFAULT_TABLE table=default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.917474 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff4e625abf14fbd94a8102dfa5fc993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.918879 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 (DEFAULT_TABLE table=default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.919283 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b665b63a7341d699d499ffc8cfd9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.926915 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:05.929329 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b0d5f8da41ce40019fd3c17f32f07b01" } new_table_name: "default.Table_3521ed8aec364fee9b3ee28e76464a11" modify_external_catalogs: true
I20251029 08:52:05.929770 1177 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.929929 1177 ts_tablet_manager.cc:1403] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20251029 08:52:05.930828 1177 raft_consensus.cc:359] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.930968 1177 raft_consensus.cc:385] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.931044 1177 raft_consensus.cc:740] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.931260 1177 consensus_queue.cc:260] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.931726 1177 ts_tablet_manager.cc:1434] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:05.931936 1177 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.932392 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet d64b1a762f124195b4369c096587881d (table default.Table_8325dc8d37f147e79a2672a42a930de9 [id=12d36151f38e4a5783f82884f1eea461]). Expected version 3 got 0
I20251029 08:52:05.933246 592 catalog_manager.cc:5649] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:05.933892 1177 tablet_bootstrap.cc:654] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.934026 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:05.934366 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:05.936852 1178 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.937157 1178 ts_tablet_manager.cc:1403] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.008s
I20251029 08:52:05.937933 1178 raft_consensus.cc:359] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.938174 1178 raft_consensus.cc:385] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.938320 1178 raft_consensus.cc:740] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.938632 1178 consensus_queue.cc:260] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.940095 1178 ts_tablet_manager.cc:1434] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:05.941151 1178 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.942745 752 tablet_service.cc:1505] Processing CreateTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 (DEFAULT_TABLE table=default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.943217 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b665b63a7341d699d499ffc8cfd9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.943364 1178 tablet_bootstrap.cc:654] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.945310 1177 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.945531 1177 ts_tablet_manager.cc:1403] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251029 08:52:05.947247 1177 raft_consensus.cc:359] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.947572 1177 raft_consensus.cc:385] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.947660 1177 raft_consensus.cc:740] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.948091 1177 consensus_queue.cc:260] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.948828 1177 ts_tablet_manager.cc:1434] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20251029 08:52:05.949148 1177 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:05.949625 1178 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.949879 1178 ts_tablet_manager.cc:1403] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20251029 08:52:05.951049 1177 tablet_bootstrap.cc:654] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.951741 1178 raft_consensus.cc:359] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.952128 1178 raft_consensus.cc:385] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.952262 1178 raft_consensus.cc:740] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.952620 1178 consensus_queue.cc:260] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.954011 1178 ts_tablet_manager.cc:1434] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:05.955277 1178 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.957394 1178 tablet_bootstrap.cc:654] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.961007 1183 tablet_bootstrap.cc:492] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:05.961303 1183 ts_tablet_manager.cc:1403] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.217s user 0.003s sys 0.002s
I20251029 08:52:05.962430 1183 raft_consensus.cc:359] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.962700 1183 raft_consensus.cc:385] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.962832 1183 raft_consensus.cc:740] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.963163 1183 consensus_queue.cc:260] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.966909 1178 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:05.967136 1178 ts_tablet_manager.cc:1403] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20251029 08:52:05.967077 892 tablet_service.cc:1505] Processing CreateTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 (DEFAULT_TABLE table=default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.967515 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b665b63a7341d699d499ffc8cfd9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.967485 1046 tablet_service.cc:1505] Processing CreateTablet for tablet ba1bcb3f6c134de2b322696501927a16 (DEFAULT_TABLE table=default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.968096 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1bcb3f6c134de2b322696501927a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.968046 1178 raft_consensus.cc:359] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.968230 1178 raft_consensus.cc:385] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.968320 1178 raft_consensus.cc:740] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.968570 1178 consensus_queue.cc:260] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.970091 1178 ts_tablet_manager.cc:1434] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:05.971040 1178 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:05.971144 1353 raft_consensus.cc:493] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.971293 1353 raft_consensus.cc:515] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.971966 1353 leader_election.cc:290] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:05.972525 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13eeb2e8951b421e9d6f8f228a956e3d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:05.972733 1069 raft_consensus.cc:2468] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:05.973593 861 leader_election.cc:304] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:05.973937 861 leader_election.cc:343] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:05.974264 1353 raft_consensus.cc:2804] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:05.974354 1353 raft_consensus.cc:493] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:05.974450 1353 raft_consensus.cc:3060] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.974615 1178 tablet_bootstrap.cc:654] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.975261 1262 fs_manager.cc:714] Time spent opening directory manager: real 0.218s user 0.003s sys 0.001s
I20251029 08:52:05.978000 1353 raft_consensus.cc:515] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.979151 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13eeb2e8951b421e9d6f8f228a956e3d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:05.979358 1069 raft_consensus.cc:3060] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:05.980875 1353 leader_election.cc:290] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:05.982406 1069 raft_consensus.cc:2468] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:05.982751 861 leader_election.cc:304] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:05.983007 1353 raft_consensus.cc:2804] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:05.983146 1353 raft_consensus.cc:697] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
W20251029 08:52:05.983593 861 leader_election.cc:343] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:05.983428 1353 consensus_queue.cc:237] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.985512 890 tablet_service.cc:1505] Processing CreateTablet for tablet ba1bcb3f6c134de2b322696501927a16 (DEFAULT_TABLE table=default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.985602 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (DEFAULT_TABLE table=default.Table_b0d5f8da41ce40019fd3c17f32f07b01 [id=e095ee6c7ffc44019821263c022eee81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.986054 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db77250e3ef489e8ab7dbcfcbb54385. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.986060 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1bcb3f6c134de2b322696501927a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.987598 1193 raft_consensus.cc:493] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:05.987735 1193 raft_consensus.cc:515] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:05.988384 1193 leader_election.cc:290] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:05.989965 861 leader_election.cc:343] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: cc1d1d5b2a9e4051bbcd25be1578ee73
W20251029 08:52:05.990222 861 leader_election.cc:343] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: cc1d1d5b2a9e4051bbcd25be1578ee73
I20251029 08:52:05.990326 861 leader_election.cc:304] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:05.990736 1193 raft_consensus.cc:2749] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:05.992153 752 tablet_service.cc:1505] Processing CreateTablet for tablet ba1bcb3f6c134de2b322696501927a16 (DEFAULT_TABLE table=default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:05.992631 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1bcb3f6c134de2b322696501927a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:05.996587 1177 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:05.996793 1177 ts_tablet_manager.cc:1403] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.003s
I20251029 08:52:05.997663 1177 raft_consensus.cc:359] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.997923 1177 raft_consensus.cc:385] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:05.998070 1177 raft_consensus.cc:740] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:05.998371 1177 consensus_queue.cc:260] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:05.999059 1177 ts_tablet_manager.cc:1434] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:05.999447 1177 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.002084 1177 tablet_bootstrap.cc:654] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.010777 1366 raft_consensus.cc:493] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.010985 1366 raft_consensus.cc:515] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.011715 1366 leader_election.cc:290] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.012217 1177 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.012377 1177 ts_tablet_manager.cc:1403] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251029 08:52:06.012845 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf4934c3842a463e857262ceed37e0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.013195 1177 raft_consensus.cc:359] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.013336 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf4934c3842a463e857262ceed37e0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.013438 1177 raft_consensus.cc:385] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.013523 1177 raft_consensus.cc:740] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.013547 925 raft_consensus.cc:2468] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.014428 718 leader_election.cc:304] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:06.014912 1366 raft_consensus.cc:2804] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.015010 1366 raft_consensus.cc:493] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.015120 1366 raft_consensus.cc:3060] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:06.016587 720 leader_election.cc:343] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.019372 1366 raft_consensus.cc:515] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.020180 1366 leader_election.cc:290] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.020521 1177 consensus_queue.cc:260] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.020884 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf4934c3842a463e857262ceed37e0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:06.021296 1177 ts_tablet_manager.cc:1434] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 08:52:06.021755 1177 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.022223 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf4934c3842a463e857262ceed37e0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:06.022435 925 raft_consensus.cc:3060] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.023720 1177 tablet_bootstrap.cc:654] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.025704 925 raft_consensus.cc:2468] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
W20251029 08:52:06.026993 720 leader_election.cc:343] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.027482 718 leader_election.cc:304] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:06.027789 1366 raft_consensus.cc:2804] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.027944 1366 raft_consensus.cc:697] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:06.028349 1366 consensus_queue.cc:237] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.028963 745 tablet_service.cc:1505] Processing CreateTablet for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (DEFAULT_TABLE table=default.Table_b0d5f8da41ce40019fd3c17f32f07b01 [id=e095ee6c7ffc44019821263c022eee81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.029457 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db77250e3ef489e8ab7dbcfcbb54385. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.035050 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 consensus state has no leader
I20251029 08:52:06.035343 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 94738d48801a4fb4a7ad1f5c4022afe1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.035748 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 28face87091b49d2971217618c95d2f5
I20251029 08:52:06.036639 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3295bb0b47714080a5c3f34049acd487
I20251029 08:52:06.037281 1178 tablet_bootstrap.cc:492] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.037456 1178 ts_tablet_manager.cc:1403] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.067s user 0.025s sys 0.002s
I20251029 08:52:06.038646 1178 raft_consensus.cc:359] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.038849 1178 raft_consensus.cc:385] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.038924 1178 raft_consensus.cc:740] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.039152 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.039183 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.040230 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:06.040338 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 28face87091b49d2971217618c95d2f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28face87091b49d2971217618c95d2f5 already in progress: creating tablet
I20251029 08:52:06.040575 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:06.040648 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3295bb0b47714080a5c3f34049acd487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3295bb0b47714080a5c3f34049acd487 already in progress: creating tablet
W20251029 08:52:06.041110 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3295bb0b47714080a5c3f34049acd487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3295bb0b47714080a5c3f34049acd487 already in progress: creating tablet
I20251029 08:52:06.040043 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:06.041549 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 28face87091b49d2971217618c95d2f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28face87091b49d2971217618c95d2f5 already in progress: creating tablet
W20251029 08:52:06.041812 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 28face87091b49d2971217618c95d2f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28face87091b49d2971217618c95d2f5 already in progress: creating tablet
I20251029 08:52:06.042222 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:06.042996 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3295bb0b47714080a5c3f34049acd487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3295bb0b47714080a5c3f34049acd487 already in progress: creating tablet
I20251029 08:52:06.043659 1177 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.045192 1177 ts_tablet_manager.cc:1403] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20251029 08:52:06.045970 1177 raft_consensus.cc:359] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.041193 1178 consensus_queue.cc:260] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.046720 1178 ts_tablet_manager.cc:1434] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:06.047307 1178 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.047755 1177 raft_consensus.cc:385] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.047878 1177 raft_consensus.cc:740] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.048409 1177 consensus_queue.cc:260] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.049650 1178 tablet_bootstrap.cc:654] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.049721 1177 ts_tablet_manager.cc:1434] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:06.050700 1177 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.052531 1177 tablet_bootstrap.cc:654] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:05.964510 1183 ts_tablet_manager.cc:1434] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:06.059180 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (DEFAULT_TABLE table=default.Table_b0d5f8da41ce40019fd3c17f32f07b01 [id=e095ee6c7ffc44019821263c022eee81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.059713 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db77250e3ef489e8ab7dbcfcbb54385. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.060290 1038 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 (DEFAULT_TABLE table=default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.060686 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a7b31ced24388bc9d9964a42fa203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.066984 1183 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:06.077214 1178 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.077450 1178 ts_tablet_manager.cc:1403] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.004s
I20251029 08:52:06.078527 745 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 (DEFAULT_TABLE table=default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.078601 1178 raft_consensus.cc:359] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.078851 1178 raft_consensus.cc:385] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.078980 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a7b31ced24388bc9d9964a42fa203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.080850 1353 raft_consensus.cc:493] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.081010 1353 raft_consensus.cc:515] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.081600 1353 leader_election.cc:290] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.080678 1177 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.081848 1177 ts_tablet_manager.cc:1403] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.020s
I20251029 08:52:06.082590 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.083058 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.083253 783 raft_consensus.cc:2468] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.083386 1177 raft_consensus.cc:359] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.083587 1177 raft_consensus.cc:385] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.083668 1177 raft_consensus.cc:740] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:06.083774 861 leader_election.cc:343] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.083949 1177 consensus_queue.cc:260] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.084079 1178 raft_consensus.cc:740] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.084335 1178 consensus_queue.cc:260] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.084494 861 leader_election.cc:304] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:06.084614 1177 ts_tablet_manager.cc:1434] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:06.084853 1178 ts_tablet_manager.cc:1434] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:06.084908 1177 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.085033 1353 raft_consensus.cc:2804] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.085129 1353 raft_consensus.cc:493] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.085156 1178 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.085230 1353 raft_consensus.cc:3060] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.086844 1178 tablet_bootstrap.cc:654] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.087080 1177 tablet_bootstrap.cc:654] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.090332 1353 raft_consensus.cc:515] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.092587 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:06.093371 861 leader_election.cc:343] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.094525 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.094766 783 raft_consensus.cc:3060] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.098637 783 raft_consensus.cc:2468] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:06.098805 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c1103092b7ae447dbfd1b7782cde6faa" } new_table_name: "default.Table_b78810487e57403d9cc5c7026930e2f2" modify_external_catalogs: true
I20251029 08:52:06.100371 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e735241cb8a04d689487db8e497b2d31" } new_table_name: "default.Table_301dd57c095940a4b34651398f5d641b" modify_external_catalogs: true
I20251029 08:52:06.108690 892 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 (DEFAULT_TABLE table=default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.109135 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a7b31ced24388bc9d9964a42fa203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.109333 589 master_service.cc:954] ReplaceTablet: received request to replace tablet a57b1dbbdf854bc6a6e1b5dc88b63bff from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.109817 589 master_service.cc:954] ReplaceTablet: received request to replace tablet e0b459b366cb402db2e7e8fee2b3e855 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.112502 861 leader_election.cc:304] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:06.112810 1193 raft_consensus.cc:2804] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.113636 1342 consensus_queue.cc:1048] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:06.116227 1353 leader_election.cc:290] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.120051 745 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.121220 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2773cae9814df79ee60ea8b0fe4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.131620 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8626877e3cfc4cb48f6caa92638b4917
I20251029 08:52:06.132215 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b273c488f33c498890f1739380b18239
I20251029 08:52:06.132630 590 catalog_manager.cc:6719] ReplaceTablet: tablet b273c488f33c498890f1739380b18239 deleted and replaced by tablet 29adc6d53af5419dbda1d41d75d37e71
I20251029 08:52:06.133545 1193 raft_consensus.cc:697] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:06.134090 1193 consensus_queue.cc:237] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.134596 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6e61b44def8f4ea283b27494c59f9926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.135126 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e61b44def8f4ea283b27494c59f9926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.137008 892 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.137791 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:06.138360 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b273c488f33c498890f1739380b18239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b273c488f33c498890f1739380b18239 already in progress: creating tablet
W20251029 08:52:06.138614 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8626877e3cfc4cb48f6caa92638b4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8626877e3cfc4cb48f6caa92638b4917 already in progress: creating tablet
I20251029 08:52:06.140175 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 14adc0f009d94287b92eda7b171c7c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.140473 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9a1d9f17776045feba0e8b4104b2fd24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.140993 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a1d9f17776045feba0e8b4104b2fd24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:06.141016 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 14adc0f009d94287b92eda7b171c7c13 because the tablet was not found. No further retry: Not found: Tablet not found: 14adc0f009d94287b92eda7b171c7c13
I20251029 08:52:06.144100 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:06.144953 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8626877e3cfc4cb48f6caa92638b4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8626877e3cfc4cb48f6caa92638b4917 already in progress: creating tablet
I20251029 08:52:06.145586 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.145588 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.145907 1176 tablet_replica.cc:333] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.146127 1176 raft_consensus.cc:2243] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:06.146240 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8626877e3cfc4cb48f6caa92638b4917 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8626877e3cfc4cb48f6caa92638b4917 already in progress: creating tablet
I20251029 08:52:06.146325 1176 raft_consensus.cc:2272] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.146524 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.147442 1176 ts_tablet_manager.cc:1916] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:06.152566 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b273c488f33c498890f1739380b18239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b273c488f33c498890f1739380b18239 already in progress: creating tablet
I20251029 08:52:06.157152 1176 ts_tablet_manager.cc:1929] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.158437 1176 log.cc:1199] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b273c488f33c498890f1739380b18239
I20251029 08:52:06.158934 1176 ts_tablet_manager.cc:1950] T b273c488f33c498890f1739380b18239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.160032 1342 raft_consensus.cc:493] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.160193 1342 raft_consensus.cc:515] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.160391 1178 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.160539 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b273c488f33c498890f1739380b18239 (table default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
I20251029 08:52:06.160594 1178 ts_tablet_manager.cc:1403] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.000s
I20251029 08:52:06.161233 1342 leader_election.cc:290] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:06.161680 996 leader_election.cc:343] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 6b8a4b96d3c748daa0f467768b2b7abd
I20251029 08:52:06.161718 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b8a4b96d3c748daa0f467768b2b7abd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.161972 783 raft_consensus.cc:2468] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:06.163077 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 57096.6us
I20251029 08:52:06.164572 1183 tablet_bootstrap.cc:654] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.165941 998 leader_election.cc:304] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:06.166445 1437 raft_consensus.cc:2804] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.166618 1437 raft_consensus.cc:493] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.166744 1437 raft_consensus.cc:3060] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.169392 1342 consensus_queue.cc:1048] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:06.171355 1441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:06.173105 1434 raft_consensus.cc:493] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.173375 1178 raft_consensus.cc:359] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.173707 1178 raft_consensus.cc:385] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.173835 1178 raft_consensus.cc:740] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.173348 1434 raft_consensus.cc:515] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.175109 1434 leader_election.cc:290] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.175495 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0b459b366cb402db2e7e8fee2b3e855" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:06.176501 861 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.177783 1183 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:06.177947 1183 ts_tablet_manager.cc:1403] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.000s
I20251029 08:52:06.175910 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0b459b366cb402db2e7e8fee2b3e855" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.178382 784 raft_consensus.cc:2468] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.178712 861 leader_election.cc:304] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:06.179159 1434 raft_consensus.cc:2804] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.179647 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a31847928db44bd4b1ebd50c5130065f (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.179749 890 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.180123 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31847928db44bd4b1ebd50c5130065f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.180244 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2773cae9814df79ee60ea8b0fe4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.184686 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 898a4f58306b453e9abca2a73082bb96 (DEFAULT_TABLE table=default.Table_25994a51dae84fef8fcd0519119cf2af [id=38aeb2e83ccc408aba7f1a5438c2f6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.184753 892 tablet_service.cc:1505] Processing CreateTablet for tablet a31847928db44bd4b1ebd50c5130065f (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.185125 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898a4f58306b453e9abca2a73082bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.185155 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31847928db44bd4b1ebd50c5130065f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.196024 747 tablet_service.cc:1505] Processing CreateTablet for tablet 898a4f58306b453e9abca2a73082bb96 (DEFAULT_TABLE table=default.Table_25994a51dae84fef8fcd0519119cf2af [id=38aeb2e83ccc408aba7f1a5438c2f6fa]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:06.196339 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2db77250e3ef489e8ab7dbcfcbb54385 consensus state has no leader
I20251029 08:52:06.196525 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898a4f58306b453e9abca2a73082bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.198846 890 tablet_service.cc:1505] Processing CreateTablet for tablet 898a4f58306b453e9abca2a73082bb96 (DEFAULT_TABLE table=default.Table_25994a51dae84fef8fcd0519119cf2af [id=38aeb2e83ccc408aba7f1a5438c2f6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.199283 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898a4f58306b453e9abca2a73082bb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.200604 745 tablet_service.cc:1505] Processing CreateTablet for tablet a31847928db44bd4b1ebd50c5130065f (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.200997 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31847928db44bd4b1ebd50c5130065f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.201884 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d (DEFAULT_TABLE table=default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.202303 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2773cae9814df79ee60ea8b0fe4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.204607 1177 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.204758 1177 ts_tablet_manager.cc:1403] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.034s
I20251029 08:52:06.205673 1177 raft_consensus.cc:359] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.179314 1434 raft_consensus.cc:493] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.206496 1177 raft_consensus.cc:385] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.206517 902 tablet_service.cc:1505] Processing CreateTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (DEFAULT_TABLE table=default.Table_90db61e4d27e4b409e948b3f33c765e9 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:06.208302 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add0605edf5c4511ae4c8d9cb7f0babb consensus state has no leader
I20251029 08:52:06.208829 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1e141ef36f5541e5bba246df7272d2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.209465 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e141ef36f5541e5bba246df7272d2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:06.210973 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9702a7d8b8304b49a250c129ecf28303 consensus state has no leader
I20251029 08:52:06.211338 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cc6f14c92a63433f90c93e8bd5956878"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.211767 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc6f14c92a63433f90c93e8bd5956878 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.211814 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (DEFAULT_TABLE table=default.Table_90db61e4d27e4b409e948b3f33c765e9 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.212090 1178 consensus_queue.cc:260] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.212280 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7a673ba4fd4abcbfc3493c38dcc257. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.212606 1178 ts_tablet_manager.cc:1434] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251029 08:52:06.212913 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 94738d48801a4fb4a7ad1f5c4022afe1
I20251029 08:52:06.213371 595 catalog_manager.cc:6719] ReplaceTablet: tablet 94738d48801a4fb4a7ad1f5c4022afe1 deleted and replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:06.213554 1178 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.213655 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb8cc7be97b8484f81dfddb6bb10e37a" } new_table_name: "default.Table_8a32e8fc09e448c1af180a8d85d9158d" modify_external_catalogs: true
I20251029 08:52:06.214763 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:06.215180 589 catalog_manager.cc:6719] ReplaceTablet: tablet e0b459b366cb402db2e7e8fee2b3e855 deleted and replaced by tablet 19c515944ecf4228a7cca63cfd3565d8
I20251029 08:52:06.215478 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_823727b7e1194e56a35d64f1d58d761a" } modify_external_catalogs: true
I20251029 08:52:06.215628 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_823727b7e1194e56a35d64f1d58d761a" } modify_external_catalogs: true
I20251029 08:52:06.216327 593 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet d64b1a762f124195b4369c096587881d (table default.Table_8325dc8d37f147e79a2672a42a930de9 [id=12d36151f38e4a5783f82884f1eea461]). Expected version 3 got 0
I20251029 08:52:06.217916 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]). Expected version 4 got 0
I20251029 08:52:06.220086 1178 tablet_bootstrap.cc:654] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.222325 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7e339c9ed26a48a2a9fe172201c5cc60
I20251029 08:52:06.222769 597 catalog_manager.cc:6719] ReplaceTablet: tablet 7e339c9ed26a48a2a9fe172201c5cc60 deleted and replaced by tablet d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:06.222877 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57a9dba12bd24aedb38524e8c99dfe84" } modify_external_catalogs: true
I20251029 08:52:06.223069 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57a9dba12bd24aedb38524e8c99dfe84" } modify_external_catalogs: true
I20251029 08:52:06.223114 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e6415f97a9d64fc996f4e571e61aba62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.223699 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6415f97a9d64fc996f4e571e61aba62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.225325 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c3267734790547088d7d0aabd176ab02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.225829 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3267734790547088d7d0aabd176ab02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.229671 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 with delete_type TABLET_DATA_DELETED (replaced by tablet d30183814f5e4f2ab9d71fe493eb801c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.229972 1176 tablet_replica.cc:333] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.230227 1176 raft_consensus.cc:2243] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.230420 1176 raft_consensus.cc:2272] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.230476 752 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c515944ecf4228a7cca63cfd3565d8) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.231351 1176 ts_tablet_manager.cc:1916] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.232880 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cc01b9de1eba414ba802605ef8240b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.233369 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9ed0a270b794e469d2fdbd0ea1d96f8 consensus state has no leader
W20251029 08:52:06.234051 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc01b9de1eba414ba802605ef8240b11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.234308 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_335bd76889fc4d0aa32a3ae0baf73890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.234880 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_335bd76889fc4d0aa32a3ae0baf73890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.233417 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9cd744dfdd12453386d05dbdb493b993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.236747 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd744dfdd12453386d05dbdb493b993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.237530 1176 ts_tablet_manager.cc:1929] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:06.237717 1176 log.cc:1199] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7e339c9ed26a48a2a9fe172201c5cc60
I20251029 08:52:06.238143 1176 ts_tablet_manager.cc:1950] T 7e339c9ed26a48a2a9fe172201c5cc60 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.238902 1366 raft_consensus.cc:493] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.239050 1366 raft_consensus.cc:515] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.239251 1437 raft_consensus.cc:515] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.239418 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7e339c9ed26a48a2a9fe172201c5cc60 (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
I20251029 08:52:06.239727 1366 leader_election.cc:290] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.239957 1193 raft_consensus.cc:493] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.240142 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7a673ba4fd4abcbfc3493c38dcc257. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.240104 1434 raft_consensus.cc:3060] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.240367 1176 tablet_replica.cc:333] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.240440 1177 raft_consensus.cc:740] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.240551 1176 raft_consensus.cc:2243] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.240710 1176 raft_consensus.cc:2272] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.241360 1176 ts_tablet_manager.cc:1916] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.241698 1262 fs_manager.cc:730] Time spent opening block manager: real 0.264s user 0.001s sys 0.002s
I20251029 08:52:06.241886 1262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:52:06.242761 1262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:52:06.244244 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.246869 1176 ts_tablet_manager.cc:1929] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.247023 1176 log.cc:1199] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:06.247396 1176 ts_tablet_manager.cc:1950] T e0b459b366cb402db2e7e8fee2b3e855 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.248589 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e0b459b366cb402db2e7e8fee2b3e855 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]) successfully deleted
I20251029 08:52:06.248719 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b8818df4a5ac46bcb3921874a943a18a" } new_table_name: "default.Table_56edb0d83a24405c93e7ae78bb8ced32" modify_external_catalogs: true
I20251029 08:52:06.248937 1178 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.249133 1178 ts_tablet_manager.cc:1403] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
W20251029 08:52:06.249490 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 94738d48801a4fb4a7ad1f5c4022afe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94738d48801a4fb4a7ad1f5c4022afe1 already in progress: creating tablet
I20251029 08:52:06.249892 1178 raft_consensus.cc:359] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.250130 1178 raft_consensus.cc:385] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.250244 1178 raft_consensus.cc:740] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.250492 1178 consensus_queue.cc:260] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.250952 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b3fef48b968b4888a66108a7e79d0ef7" } modify_external_catalogs: true
I20251029 08:52:06.251116 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b3fef48b968b4888a66108a7e79d0ef7" } modify_external_catalogs: true
I20251029 08:52:06.253814 1178 ts_tablet_manager.cc:1434] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 08:52:06.254139 1178 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.254690 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b8a4b96d3c748daa0f467768b2b7abd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.256114 784 raft_consensus.cc:3060] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.258147 1193 raft_consensus.cc:515] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.259065 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.259109 1183 raft_consensus.cc:359] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.259461 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (DEFAULT_TABLE table=default.Table_90db61e4d27e4b409e948b3f33c765e9 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.259089 1437 leader_election.cc:290] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.259883 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7a673ba4fd4abcbfc3493c38dcc257. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.261497 720 leader_election.cc:343] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.263015 784 raft_consensus.cc:2468] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:06.263607 745 tablet_service.cc:1505] Processing CreateTablet for tablet e5fa181849474a559701fdb8c18d398b (DEFAULT_TABLE table=default.Table_7e3d3e525fa34d06bb82e54fc6cc95af [id=ca079b1870a64ab69697d5255756c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.264070 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fa181849474a559701fdb8c18d398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.264748 747 tablet_service.cc:1505] Processing CreateTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 (DEFAULT_TABLE table=default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.265101 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e50018f414f4b88e0e5d90cb7c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.265115 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.265609 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:06.267405 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f33c0bbf104046ccb86ff313f08da22a consensus state has no leader
I20251029 08:52:06.267812 752 tablet_service.cc:1505] Processing CreateTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 (DEFAULT_TABLE table=default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.268311 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20f26d0f7874ccb960ffe4d27e854e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.271575 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d71bb6e219494f90897bd2f8844f599e" } modify_external_catalogs: true
I20251029 08:52:06.271749 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d71bb6e219494f90897bd2f8844f599e" } modify_external_catalogs: true
I20251029 08:52:06.272851 749 tablet_service.cc:1505] Processing CreateTablet for tablet f33c0bbf104046ccb86ff313f08da22a (DEFAULT_TABLE table=default.Table_56edb0d83a24405c93e7ae78bb8ced32 [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.273228 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33c0bbf104046ccb86ff313f08da22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.275871 1051 tablet_service.cc:1505] Processing CreateTablet for tablet e5fa181849474a559701fdb8c18d398b (DEFAULT_TABLE table=default.Table_7e3d3e525fa34d06bb82e54fc6cc95af [id=ca079b1870a64ab69697d5255756c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.276463 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.276743 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c515944ecf4228a7cca63cfd3565d8) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.278101 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 with delete_type TABLET_DATA_DELETED (replaced by tablet d30183814f5e4f2ab9d71fe493eb801c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.278477 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 (DEFAULT_TABLE table=default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.279264 1183 raft_consensus.cc:385] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.279654 998 leader_election.cc:304] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:06.279934 1437 raft_consensus.cc:2804] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.280797 1437 raft_consensus.cc:697] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
W20251029 08:52:06.283970 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e0b459b366cb402db2e7e8fee2b3e855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b459b366cb402db2e7e8fee2b3e855 already in progress: creating tablet
I20251029 08:52:06.284080 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8daa66608cf94913afb94ef661b3c898
I20251029 08:52:06.284500 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fa181849474a559701fdb8c18d398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.285634 1183 raft_consensus.cc:740] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.286513 1038 tablet_service.cc:1505] Processing CreateTablet for tablet f33c0bbf104046ccb86ff313f08da22a (DEFAULT_TABLE table=default.Table_56edb0d83a24405c93e7ae78bb8ced32 [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.286659 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:06.286815 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.286989 1179 tablet_replica.cc:333] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:06.286983 588 catalog_manager.cc:5649] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:06.286799 1437 consensus_queue.cc:237] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.287599 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
W20251029 08:52:06.287638 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8daa66608cf94913afb94ef661b3c898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8daa66608cf94913afb94ef661b3c898 already in progress: creating tablet
I20251029 08:52:06.287954 593 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47] alter complete (version 4)
I20251029 08:52:06.288512 1179 raft_consensus.cc:2243] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:06.288828 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:06.288868 1179 raft_consensus.cc:2272] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.290944 1178 tablet_bootstrap.cc:654] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.291153 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:06.273098 1177 consensus_queue.cc:260] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.295032 1179 ts_tablet_manager.cc:1916] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.295445 752 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.295636 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c515944ecf4228a7cca63cfd3565d8) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.295873 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.295343 1043 tablet_service.cc:1505] Processing CreateTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 (DEFAULT_TABLE table=default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.298652 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet e899e183e255498b83a6d740af98de42 with delete_type TABLET_DATA_DELETED (replaced by tablet 904523f7164849139cddb1100e95b824) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.301105 1176 tablet_replica.cc:333] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.302668 1176 raft_consensus.cc:2243] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.302842 1176 raft_consensus.cc:2272] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.303704 1176 ts_tablet_manager.cc:1916] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.304113 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 06eb710809754e7abd00a345e8ae591a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.304298 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.304342 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.304876 1433 raft_consensus.cc:493] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.305071 1433 raft_consensus.cc:515] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.305631 749 tablet_service.cc:1505] Processing CreateTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 (DEFAULT_TABLE table=default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.305836 1433 leader_election.cc:290] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.306154 748 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.306427 747 tablet_service.cc:1505] Processing CreateTablet for tablet e4f80728a7a04583bada485707beb806 (DEFAULT_TABLE table=default.Table_54314bbc9f794709b664977bd9f32a93 [id=99ae7a11d37346e18a01a330d7fa42b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.306739 1193 leader_election.cc:290] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.307089 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c706343d44a4807b1ea60c1b1e1ea49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.307139 1177 ts_tablet_manager.cc:1434] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.093s user 0.001s sys 0.001s
I20251029 08:52:06.307448 1177 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.309243 1177 tablet_bootstrap.cc:654] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.310151 1353 raft_consensus.cc:493] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.311035 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2c706343d44a4807b1ea60c1b1e1ea49
W20251029 08:52:06.311311 861 leader_election.cc:343] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:06.311950 1353 raft_consensus.cc:515] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.313688 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "933817bf2058464f837ba96921c4ed63" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.313901 784 raft_consensus.cc:2468] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.314884 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.316244 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.316417 784 raft_consensus.cc:2468] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.316586 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet db9cf2a8f2254a38a725bcd3c83e8509 with delete_type TABLET_DATA_DELETED (replaced by tablet c8cb0c6c6a1c4877a89476d4822f48db) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.318579 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "933817bf2058464f837ba96921c4ed63" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.318921 1042 tablet_service.cc:1505] Processing CreateTablet for tablet e4f80728a7a04583bada485707beb806 (DEFAULT_TABLE table=default.Table_54314bbc9f794709b664977bd9f32a93 [id=99ae7a11d37346e18a01a330d7fa42b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.320117 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.320528 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 (DEFAULT_TABLE table=default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.321477 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.321651 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f80728a7a04583bada485707beb806. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.322566 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.323741 745 tablet_service.cc:1505] Processing CreateTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe (DEFAULT_TABLE table=default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.324898 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e50018f414f4b88e0e5d90cb7c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.325381 1353 leader_election.cc:290] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.326081 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.326419 1045 tablet_service.cc:1505] Processing CreateTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe (DEFAULT_TABLE table=default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.326671 861 leader_election.cc:304] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:06.327214 861 leader_election.cc:343] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:06.327340 861 leader_election.cc:304] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:06.328270 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572c68b32d443b2a019e1f4bf2d82fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.326769 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:06.329778 861 leader_election.cc:343] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.330168 861 leader_election.cc:304] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:06.330556 1433 raft_consensus.cc:2804] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.330698 1433 raft_consensus.cc:493] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.330804 1433 raft_consensus.cc:3060] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.330965 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572c68b32d443b2a019e1f4bf2d82fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.323534 752 tablet_service.cc:1505] Processing CreateTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.332883 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.333096 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf (DEFAULT_TABLE table=default.Table_685d1d31eb774418a8e997595f7e8ff8 [id=93bd45834be24532b19ed170c43ac0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.333328 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.335203 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.336278 1183 consensus_queue.cc:260] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.336484 758 tablet_service.cc:1505] Processing CreateTablet for tablet e626994e96d54c0d913fdedc85fd1155 (DEFAULT_TABLE table=default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.337509 762 tablet_service.cc:1505] Processing CreateTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 (DEFAULT_TABLE table=default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.339134 746 tablet_service.cc:1505] Processing CreateTablet for tablet fd25eeb89e574297a92d8e978a79d768 (DEFAULT_TABLE table=default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.339350 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:06.343398 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc6d53af5419dbda1d41d75d37e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.343747 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.343410 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b8a4b96d3c748daa0f467768b2b7abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:02 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.346022 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33c0bbf104046ccb86ff313f08da22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.346050 1353 raft_consensus.cc:2804] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.346944 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ab2c1cce4615467595cb78eb6ca1b4ec" } modify_external_catalogs: true
I20251029 08:52:06.347265 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ab2c1cce4615467595cb78eb6ca1b4ec" } modify_external_catalogs: true
I20251029 08:52:06.348811 1353 raft_consensus.cc:493] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.348951 1353 raft_consensus.cc:3060] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.349473 1193 raft_consensus.cc:2749] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.349442 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf (DEFAULT_TABLE table=default.Table_685d1d31eb774418a8e997595f7e8ff8 [id=93bd45834be24532b19ed170c43ac0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.349891 756 tablet_service.cc:1505] Processing CreateTablet for tablet 07906300f1d54d4e8fde5165ee14c44f (DEFAULT_TABLE table=default.Table_335bd76889fc4d0aa32a3ae0baf73890 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.349906 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.350885 1434 raft_consensus.cc:515] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.351107 750 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.351303 759 tablet_service.cc:1505] Processing CreateTablet for tablet e1e2be77182e4259b76d20cae4e9e478 (DEFAULT_TABLE table=default.Table_cc6f14c92a63433f90c93e8bd5956878 [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.351557 1434 leader_election.cc:290] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.351696 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9f0b579ebd4d2db813459611d7f9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.352596 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.353407 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 (DEFAULT_TABLE table=default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.355182 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc6d53af5419dbda1d41d75d37e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.355858 751 tablet_service.cc:1505] Processing CreateTablet for tablet d896959a4c684e1b8b68a36125e751b9 (DEFAULT_TABLE table=default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.356113 760 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.356874 755 tablet_service.cc:1505] Processing CreateTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.357221 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608b7095d42941ccb0c4eb57f6754e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.359273 757 tablet_service.cc:1505] Processing CreateTablet for tablet bc9c221eaa9a4721ba25761206a13645 (DEFAULT_TABLE table=default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.359972 761 tablet_service.cc:1505] Processing CreateTablet for tablet c7024e6eb4124109ad1575d216739fe2 (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.360270 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626994e96d54c0d913fdedc85fd1155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.361877 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608b7095d42941ccb0c4eb57f6754e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.362622 1183 ts_tablet_manager.cc:1434] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.177s user 0.002s sys 0.000s
W20251029 08:52:06.363356 861 leader_election.cc:343] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.365761 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd25eeb89e574297a92d8e978a79d768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.366247 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e626994e96d54c0d913fdedc85fd1155 (DEFAULT_TABLE table=default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.366406 750 tablet_service.cc:1505] Processing CreateTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe (DEFAULT_TABLE table=default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.367244 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e2be77182e4259b76d20cae4e9e478. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.368628 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 5385fef404a4463595074f0c8ff822d7 (DEFAULT_TABLE table=default.Table_6e61b44def8f4ea283b27494c59f9926 [id=2cfb2b85354d43e7a203d35a4448346d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.368707 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe (DEFAULT_TABLE table=default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.369084 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.369765 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d896959a4c684e1b8b68a36125e751b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.370040 1035 tablet_service.cc:1505] Processing CreateTablet for tablet e1e2be77182e4259b76d20cae4e9e478 (DEFAULT_TABLE table=default.Table_cc6f14c92a63433f90c93e8bd5956878 [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.362166 764 tablet_service.cc:1505] Processing CreateTablet for tablet 5385fef404a4463595074f0c8ff822d7 (DEFAULT_TABLE table=default.Table_6e61b44def8f4ea283b27494c59f9926 [id=2cfb2b85354d43e7a203d35a4448346d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.370844 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0b459b366cb402db2e7e8fee2b3e855" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:06.370900 1183 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:06.371059 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fcf8a3e0f4770a9743fddffa6495f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.371745 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07906300f1d54d4e8fde5165ee14c44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.372373 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9f0b579ebd4d2db813459611d7f9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.373754 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5385fef404a4463595074f0c8ff822d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.375664 861 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:06.377219 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7024e6eb4124109ad1575d216739fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.378710 1033 tablet_service.cc:1505] Processing CreateTablet for tablet bc9c221eaa9a4721ba25761206a13645 (DEFAULT_TABLE table=default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.378778 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 07906300f1d54d4e8fde5165ee14c44f (DEFAULT_TABLE table=default.Table_335bd76889fc4d0aa32a3ae0baf73890 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.385267 753 tablet_service.cc:1505] Processing CreateTablet for tablet d30183814f5e4f2ab9d71fe493eb801c (DEFAULT_TABLE table=default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.385910 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9c221eaa9a4721ba25761206a13645. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.386651 760 tablet_service.cc:1505] Processing CreateTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.387300 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787d6d7f1870450fa21cb9cba34e72fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.387822 749 tablet_service.cc:1505] Processing CreateTablet for tablet 072cf27b0743488c8166e03b5e61c545 (DEFAULT_TABLE table=default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.389261 1353 raft_consensus.cc:515] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.392493 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5385fef404a4463595074f0c8ff822d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.392748 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07906300f1d54d4e8fde5165ee14c44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.378430 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c7024e6eb4124109ad1575d216739fe2 (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.393350 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9c221eaa9a4721ba25761206a13645. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.394330 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fcf8a3e0f4770a9743fddffa6495f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.395367 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7024e6eb4124109ad1575d216739fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.395908 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787d6d7f1870450fa21cb9cba34e72fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.396157 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072cf27b0743488c8166e03b5e61c545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.400591 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c706343d44a4807b1ea60c1b1e1ea49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.401058 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20f26d0f7874ccb960ffe4d27e854e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.401460 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f80728a7a04583bada485707beb806. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.402269 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626994e96d54c0d913fdedc85fd1155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.403524 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_56edb0d83a24405c93e7ae78bb8ced32" } new_table_name: "default.Table_e6dc65fd6a3f4d53a1c27c9fecf32490" modify_external_catalogs: true
I20251029 08:52:06.405449 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2723cdd513ee4e8ca6773b38f785b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.408408 1049 tablet_service.cc:1505] Processing CreateTablet for tablet fd25eeb89e574297a92d8e978a79d768 (DEFAULT_TABLE table=default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.408560 763 tablet_service.cc:1505] Processing CreateTablet for tablet 840ff254244f43a18679232279e02d1f (DEFAULT_TABLE table=default.Table_c3267734790547088d7d0aabd176ab02 [id=383565dcefdd479288ddfc261cbced6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.409049 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd25eeb89e574297a92d8e978a79d768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.407496 754 tablet_service.cc:1505] Processing CreateTablet for tablet 19c515944ecf4228a7cca63cfd3565d8 (DEFAULT_TABLE table=default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.410426 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30183814f5e4f2ab9d71fe493eb801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.412290 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840ff254244f43a18679232279e02d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.412904 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c515944ecf4228a7cca63cfd3565d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.414959 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 57e2b19a227740a28ae601d77e62597b
W20251029 08:52:06.426342 861 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.426672 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e2be77182e4259b76d20cae4e9e478. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.435611 1035 tablet_service.cc:1505] Processing CreateTablet for tablet d896959a4c684e1b8b68a36125e751b9 (DEFAULT_TABLE table=default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.440075 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d896959a4c684e1b8b68a36125e751b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.456120 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 072cf27b0743488c8166e03b5e61c545 (DEFAULT_TABLE table=default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.456568 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072cf27b0743488c8166e03b5e61c545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.468508 1366 raft_consensus.cc:493] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.468704 1366 raft_consensus.cc:515] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.469419 1366 leader_election.cc:290] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.470723 1176 ts_tablet_manager.cc:1929] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.470866 1176 log.cc:1199] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a61235da8c7a4f31b29d5d50c53c347d
I20251029 08:52:06.471275 1176 ts_tablet_manager.cc:1950] T a61235da8c7a4f31b29d5d50c53c347d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.474067 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.485807 1353 leader_election.cc:290] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.443905 861 leader_election.cc:304] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:06.486671 1177 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.487488 1433 raft_consensus.cc:515] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.488924 1434 raft_consensus.cc:493] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.490338 1353 raft_consensus.cc:2749] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.490546 1177 ts_tablet_manager.cc:1403] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.179s user 0.002s sys 0.006s
I20251029 08:52:06.491283 1177 raft_consensus.cc:359] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.492542 1434 raft_consensus.cc:515] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.491941 1433 leader_election.cc:290] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.496409 1176 tablet_replica.cc:333] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.496606 1176 raft_consensus.cc:2243] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.496783 1176 raft_consensus.cc:2272] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.497347 1176 ts_tablet_manager.cc:1916] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:06.504897 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 112158us
I20251029 08:52:06.506117 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "933817bf2058464f837ba96921c4ed63" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.506352 784 raft_consensus.cc:3060] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.509249 784 raft_consensus.cc:2468] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:06.509570 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.509742 783 raft_consensus.cc:3060] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.513767 1177 raft_consensus.cc:385] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.514279 1178 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.514465 1178 ts_tablet_manager.cc:1403] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.260s user 0.008s sys 0.020s
I20251029 08:52:06.515424 783 raft_consensus.cc:2468] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:06.515929 1178 raft_consensus.cc:359] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.516144 1178 raft_consensus.cc:385] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.516223 1178 raft_consensus.cc:740] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.516270 1177 raft_consensus.cc:740] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.514662 861 leader_election.cc:304] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:06.517519 861 leader_election.cc:343] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:06.517428 1178 consensus_queue.cc:260] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.517735 861 leader_election.cc:304] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:06.517998 1178 ts_tablet_manager.cc:1434] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20251029 08:52:06.518896 861 leader_election.cc:343] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:06.519007 861 leader_election.cc:304] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:06.519243 1193 raft_consensus.cc:2749] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.519418 1353 raft_consensus.cc:2804] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.520088 1176 ts_tablet_manager.cc:1929] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.520253 1176 log.cc:1199] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/06eb710809754e7abd00a345e8ae591a
I20251029 08:52:06.520395 1353 raft_consensus.cc:697] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:06.520670 1353 consensus_queue.cc:237] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.522969 1433 raft_consensus.cc:2804] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.523097 1433 raft_consensus.cc:697] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:06.524437 1176 ts_tablet_manager.cc:1950] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.526865 1178 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:06.416973 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f33c0bbf104046ccb86ff313f08da22a consensus state has no leader
I20251029 08:52:06.425925 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_301dd57c095940a4b34651398f5d641b" } new_table_name: "default.Table_3d5e7e9f16c04071af70d3a47624570d" modify_external_catalogs: true
I20251029 08:52:06.426399 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dc4e52894b5141c0b0297a471a1c6eef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.528467 1176 tablet_replica.cc:333] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:06.528591 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc4e52894b5141c0b0297a471a1c6eef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.528695 1176 raft_consensus.cc:2243] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.528836 1176 raft_consensus.cc:2272] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.442368 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1f17f82721d4c14ac66469353b2bcc9" } new_table_name: "default.Table_7b45e52828bf4727815bfbb7c1ddb9c3" modify_external_catalogs: true
I20251029 08:52:06.530603 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6f454ac2adfa4a4da06fa1180ca26c58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.531069 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f454ac2adfa4a4da06fa1180ca26c58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.442806 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4f1f60e2c98f47079dc3996b94e0d4c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.531061 1177 consensus_queue.cc:260] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:06.531563 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.531785 1176 ts_tablet_manager.cc:1916] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.443135 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2e2db23cf2c6466b9faa928d7adac713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.532004 1434 leader_election.cc:290] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:06.532361 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e2db23cf2c6466b9faa928d7adac713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.443480 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_67b2c204ba81497680343eaa3b48b255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.532748 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67b2c204ba81497680343eaa3b48b255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.443743 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_55cdb37e69164060ae6595fe21e9db73" } new_table_name: "default.Table_dc9e5c781bc0463d9df56e2db965b1bb" modify_external_catalogs: true
I20251029 08:52:06.524057 1433 consensus_queue.cc:237] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.535953 1178 tablet_bootstrap.cc:654] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.472286 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8642fe3704384eaa803ad55dcdcdf6b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.500509 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a61235da8c7a4f31b29d5d50c53c347d (table default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]) successfully deleted
W20251029 08:52:06.536319 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e0b459b366cb402db2e7e8fee2b3e855 because the tablet was not found. No further retry: Not found: Tablet not found: e0b459b366cb402db2e7e8fee2b3e855
W20251029 08:52:06.536370 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8642fe3704384eaa803ad55dcdcdf6b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.536762 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 06eb710809754e7abd00a345e8ae591a (table default.Table_7a0da910696e45fd9559acf90e71cde8 [id=f9f88be350174f47aef46005393b6578]) successfully deleted
I20251029 08:52:06.537846 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.542003 1177 ts_tablet_manager.cc:1434] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20251029 08:52:06.549888 1177 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.551069 1176 ts_tablet_manager.cc:1929] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.551757 1366 raft_consensus.cc:493] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.552039 1177 tablet_bootstrap.cc:654] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.552089 1366 raft_consensus.cc:515] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.552816 1366 leader_election.cc:290] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.553478 1176 log.cc:1199] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:06.554059 1176 ts_tablet_manager.cc:1950] T 16a2b37e45e444caa3f5117ae9a25304 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:06.555294 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a756d6b5b3648e88cd2bd3400e51bed consensus state has no leader
I20251029 08:52:06.555853 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 16a2b37e45e444caa3f5117ae9a25304 (table default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]) successfully deleted
I20251029 08:52:06.556995 1178 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.557143 1178 ts_tablet_manager.cc:1403] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20251029 08:52:06.557957 1178 raft_consensus.cc:359] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.558135 1178 raft_consensus.cc:385] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.558224 1178 raft_consensus.cc:740] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.558458 1178 consensus_queue.cc:260] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.562914 1178 ts_tablet_manager.cc:1434] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:06.566201 1178 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.568408 1178 tablet_bootstrap.cc:654] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.569411 1176 tablet_replica.cc:333] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.569737 1176 raft_consensus.cc:2243] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.569999 1176 raft_consensus.cc:2272] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.570633 1176 ts_tablet_manager.cc:1916] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.571959 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_19667d766eb94067b16badc787433f2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.572700 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19667d766eb94067b16badc787433f2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.575165 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_03feaca359af4f749d12bfa146dc1859" } new_table_name: "default.Table_bc205fa7964840f9802b4bb2a9a4f954" modify_external_catalogs: true
I20251029 08:52:06.576794 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_00d50b5cd5d045828ba7b366df301214"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.577459 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00d50b5cd5d045828ba7b366df301214 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.578384 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.578665 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e808bff7558472099587a0f95c002ff" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.578908 923 raft_consensus.cc:2468] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.579636 902 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c515944ecf4228a7cca63cfd3565d8) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.579945 1175 tablet_replica.cc:333] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:06.580160 1175 raft_consensus.cc:2243] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.580288 892 tablet_service.cc:1505] Processing CreateTablet for tablet e5fa181849474a559701fdb8c18d398b (DEFAULT_TABLE table=default.Table_7e3d3e525fa34d06bb82e54fc6cc95af [id=ca079b1870a64ab69697d5255756c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.580372 1175 raft_consensus.cc:2272] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.580710 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fa181849474a559701fdb8c18d398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.581275 1175 ts_tablet_manager.cc:1916] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.582134 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fa76b534de514a8cb76768b632ed61de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.582849 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa76b534de514a8cb76768b632ed61de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.580711 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e339c9ed26a48a2a9fe172201c5cc60 with delete_type TABLET_DATA_DELETED (replaced by tablet d30183814f5e4f2ab9d71fe493eb801c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.585711 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bb18d609b7424f67a50733661054820d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.586223 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb18d609b7424f67a50733661054820d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.591020 1175 ts_tablet_manager.cc:1929] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.591625 1178 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.591876 1178 ts_tablet_manager.cc:1403] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20251029 08:52:06.592757 1178 raft_consensus.cc:359] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:06.593036 1178 raft_consensus.cc:385] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.593158 1178 raft_consensus.cc:740] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.593568 1178 consensus_queue.cc:260] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:06.591856 902 tablet_service.cc:1505] Processing CreateTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 (DEFAULT_TABLE table=default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.593894 894 tablet_service.cc:1505] Processing CreateTablet for tablet e4f80728a7a04583bada485707beb806 (DEFAULT_TABLE table=default.Table_54314bbc9f794709b664977bd9f32a93 [id=99ae7a11d37346e18a01a330d7fa42b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.594089 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20f26d0f7874ccb960ffe4d27e854e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.594265 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f80728a7a04583bada485707beb806. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.592413 895 tablet_service.cc:1505] Processing CreateTablet for tablet f33c0bbf104046ccb86ff313f08da22a (DEFAULT_TABLE table=default.Table_56edb0d83a24405c93e7ae78bb8ced32 [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.592752 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.599221 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33c0bbf104046ccb86ff313f08da22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.593207 892 tablet_service.cc:1505] Processing CreateTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 (DEFAULT_TABLE table=default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.599808 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e50018f414f4b88e0e5d90cb7c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.593441 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.604511 1178 ts_tablet_manager.cc:1434] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 08:52:06.608868 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.609965 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.610261 1178 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.611786 1175 log.cc:1199] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:06.612986 1178 tablet_bootstrap.cc:654] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.614681 1176 ts_tablet_manager.cc:1929] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.614876 1176 log.cc:1199] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:06.615301 1176 ts_tablet_manager.cc:1950] T db9cf2a8f2254a38a725bcd3c83e8509 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.617378 1176 tablet_replica.cc:333] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.617807 1176 raft_consensus.cc:2243] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.618381 1176 raft_consensus.cc:2272] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.619086 1176 ts_tablet_manager.cc:1916] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.619102 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet db9cf2a8f2254a38a725bcd3c83e8509 (table default.Table_cd11e136b8054bdfb9c12527b0fc5777 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
W20251029 08:52:06.619395 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b273c488f33c498890f1739380b18239 because the tablet was not found. No further retry: Not found: Tablet not found: b273c488f33c498890f1739380b18239
W20251029 08:52:06.619606 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 39dc5ebeb36043e0aa5955bacd06c2ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39dc5ebeb36043e0aa5955bacd06c2ef already in progress: creating tablet
W20251029 08:52:06.619827 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2c706343d44a4807b1ea60c1b1e1ea49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c706343d44a4807b1ea60c1b1e1ea49 already in progress: creating tablet
I20251029 08:52:06.617175 1448 raft_consensus.cc:493] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.621990 1448 raft_consensus.cc:515] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.506295 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "933817bf2058464f837ba96921c4ed63" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:06.506582 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:06.507879 996 leader_election.cc:343] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 6b8a4b96d3c748daa0f467768b2b7abd
W20251029 08:52:06.623574 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 115686us
I20251029 08:52:06.623960 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "904523f7164849139cddb1100e95b824" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.624883 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2723cdd513ee4e8ca6773b38f785b09e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.625568 861 leader_election.cc:343] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:06.625923 861 leader_election.cc:343] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 904523f7164849139cddb1100e95b824
W20251029 08:52:06.625970 584 catalog_manager.cc:5058] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): alter failed for tablet 7e339c9ed26a48a2a9fe172201c5cc60 (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
W20251029 08:52:06.626139 861 leader_election.cc:343] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:06.626444 861 leader_election.cc:343] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.626461 1437 raft_consensus.cc:493] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.626570 861 leader_election.cc:304] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:06.626616 1437 raft_consensus.cc:515] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.626873 1434 raft_consensus.cc:2749] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.623515 1448 leader_election.cc:290] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.627278 1437 leader_election.cc:290] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:06.627945 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.628481 1070 raft_consensus.cc:3060] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.628750 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e808bff7558472099587a0f95c002ff" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:06.629405 720 leader_election.cc:343] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.633818 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.638442 1176 ts_tablet_manager.cc:1929] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.641147 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ff3274c7104c73bd35768c229833b1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.641317 1460 tablet.cc:1722] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:06.641577 1176 log.cc:1199] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4e808bff7558472099587a0f95c002ff
I20251029 08:52:06.642328 1176 ts_tablet_manager.cc:1950] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.643790 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4e808bff7558472099587a0f95c002ff (table default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]) successfully deleted
I20251029 08:52:06.644883 1176 tablet_replica.cc:333] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.649339 1176 raft_consensus.cc:2243] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.649997 1176 raft_consensus.cc:2272] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.650955 1448 consensus_queue.cc:1048] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:06.651634 1176 ts_tablet_manager.cc:1916] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.652714 902 tablet_service.cc:1505] Processing CreateTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 (DEFAULT_TABLE table=default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.653131 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c706343d44a4807b1ea60c1b1e1ea49. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.651929 720 leader_election.cc:343] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 06eb710809754e7abd00a345e8ae591a
I20251029 08:52:06.653960 1179 ts_tablet_manager.cc:1929] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:06.654155 1179 log.cc:1199] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7e339c9ed26a48a2a9fe172201c5cc60
I20251029 08:52:06.654300 894 tablet_service.cc:1505] Processing CreateTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 (DEFAULT_TABLE table=default.Table_55cdb37e69164060ae6595fe21e9db73 [id=b5a04719c6664c87a647be7fed35ca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.654696 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc6d53af5419dbda1d41d75d37e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.654832 996 leader_election.cc:304] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:06.655118 1437 raft_consensus.cc:2749] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.656198 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 840ff254244f43a18679232279e02d1f (DEFAULT_TABLE table=default.Table_c3267734790547088d7d0aabd176ab02 [id=383565dcefdd479288ddfc261cbced6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.654773 1179 ts_tablet_manager.cc:1950] T 7e339c9ed26a48a2a9fe172201c5cc60 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:06.656549 895 tablet_service.cc:1505] Processing CreateTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe (DEFAULT_TABLE table=default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.656909 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572c68b32d443b2a019e1f4bf2d82fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.657512 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7e339c9ed26a48a2a9fe172201c5cc60 (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
W20251029 08:52:06.658077 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29adc6d53af5419dbda1d41d75d37e71 consensus state has no leader
I20251029 08:52:06.661391 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d7165655d9634a7bb270da6738b6c0c5" } modify_external_catalogs: true
I20251029 08:52:06.661633 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d7165655d9634a7bb270da6738b6c0c5" } modify_external_catalogs: true
I20251029 08:52:06.662175 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6d2b9c809be24c519e176adfd43dfa45" } modify_external_catalogs: true
W20251029 08:52:06.662263 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9702a7d8b8304b49a250c129ecf28303 consensus state has no leader
I20251029 08:52:06.662346 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6d2b9c809be24c519e176adfd43dfa45" } modify_external_catalogs: true
I20251029 08:52:06.662588 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_742ee97669df417aba70fdcea4983c94" } modify_external_catalogs: true
I20251029 08:52:06.662776 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_742ee97669df417aba70fdcea4983c94" } modify_external_catalogs: true
I20251029 08:52:06.663935 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8325dc8d37f147e79a2672a42a930de9" } modify_external_catalogs: true
I20251029 08:52:06.664129 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8325dc8d37f147e79a2672a42a930de9" } modify_external_catalogs: true
I20251029 08:52:06.665160 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 63b08840d0fa40428c61a05b277ad750 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.665199 1366 raft_consensus.cc:493] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.665393 1366 raft_consensus.cc:515] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.666080 1366 leader_election.cc:290] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:06.669775 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 94738d48801a4fb4a7ad1f5c4022afe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94738d48801a4fb4a7ad1f5c4022afe1 already in progress: creating tablet
I20251029 08:52:06.509618 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d30183814f5e4f2ab9d71fe493eb801c (DEFAULT_TABLE table=default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:06.673529 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8daa66608cf94913afb94ef661b3c898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8daa66608cf94913afb94ef661b3c898 already in progress: creating tablet
I20251029 08:52:06.676041 1175 ts_tablet_manager.cc:1950] T e0b459b366cb402db2e7e8fee2b3e855 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:06.517206 1438 raft_consensus.cc:493] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.522310 1183 tablet_bootstrap.cc:654] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.676214 1438 raft_consensus.cc:515] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.676613 1175 tablet_replica.cc:333] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:06.676824 1175 raft_consensus.cc:2243] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.677012 1175 raft_consensus.cc:2272] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.677251 1438 leader_election.cc:290] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.541492 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 19c515944ecf4228a7cca63cfd3565d8 (DEFAULT_TABLE table=default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.677973 893 tablet_service.cc:1505] Processing CreateTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf (DEFAULT_TABLE table=default.Table_685d1d31eb774418a8e997595f7e8ff8 [id=93bd45834be24532b19ed170c43ac0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.678365 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9f0b579ebd4d2db813459611d7f9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.678635 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.671676 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:06.679165 720 leader_election.cc:343] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.673502 1179 tablet_replica.cc:333] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:06.679900 1179 raft_consensus.cc:2243] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.682031 1179 raft_consensus.cc:2272] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.673568 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840ff254244f43a18679232279e02d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.685747 757 tablet_service.cc:1505] Processing CreateTablet for tablet 8991ea78321545be85d5c480841e4751 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.686162 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8991ea78321545be85d5c480841e4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.688172 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c515944ecf4228a7cca63cfd3565d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.688423 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30183814f5e4f2ab9d71fe493eb801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.689285 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_cd11e136b8054bdfb9c12527b0fc5777 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 1 got 0
I20251029 08:52:06.691201 588 master_service.cc:954] ReplaceTablet: received request to replace tablet c9a1d5e1f68e4f0e8b78641bbdc98123 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.691350 890 tablet_service.cc:1505] Processing CreateTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 (DEFAULT_TABLE table=default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.691787 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608b7095d42941ccb0c4eb57f6754e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.696431 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.697007 1262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:52:06.698168 1262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:52:06.698508 1262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:52:06.700021 1179 ts_tablet_manager.cc:1916] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.700253 893 tablet_service.cc:1505] Processing CreateTablet for tablet e626994e96d54c0d913fdedc85fd1155 (DEFAULT_TABLE table=default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.700721 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626994e96d54c0d913fdedc85fd1155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.702478 591 master_service.cc:954] ReplaceTablet: received request to replace tablet a8c5008d9df9419bb4b6d678c6dc27a6 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.706584 896 tablet_service.cc:1552] Processing DeleteTablet for tablet cc1d1d5b2a9e4051bbcd25be1578ee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.707056 896 tablet_service.cc:1552] Processing DeleteTablet for tablet b4ceeaba655542dab2556d3988b6e841 with delete_type TABLET_DATA_DELETED (replaced by tablet 55842c18dbef4513a61dd9629f2e5a4c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.707476 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 453911c5a29f43e7a21fa2c47ef264e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.707983 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.708379 896 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.708734 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.709064 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.709542 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.710908 896 tablet_service.cc:1505] Processing CreateTablet for tablet 07906300f1d54d4e8fde5165ee14c44f (DEFAULT_TABLE table=default.Table_335bd76889fc4d0aa32a3ae0baf73890 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.711318 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07906300f1d54d4e8fde5165ee14c44f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.719822 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61b07eb4befa41f3a1b5f2e9ebc2277c consensus state has no leader
I20251029 08:52:06.720163 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.720906 1175 ts_tablet_manager.cc:1916] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:06.729168 996 leader_election.cc:343] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:06.732002 1177 tablet_bootstrap.cc:492] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:06.732183 1177 ts_tablet_manager.cc:1403] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.005s
W20251029 08:52:06.732185 998 leader_election.cc:343] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:06.732451 998 leader_election.cc:304] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:06.732681 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 50622.6us
I20251029 08:52:06.732897 1438 raft_consensus.cc:2749] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.732998 1177 raft_consensus.cc:359] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.733199 1177 raft_consensus.cc:385] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.733299 1177 raft_consensus.cc:740] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.733556 1177 consensus_queue.cc:260] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.734951 1177 ts_tablet_manager.cc:1434] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:06.735265 1177 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:06.737522 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet abc307b14aff4c5e937a5151aab1dfb9
W20251029 08:52:06.739060 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 57095.8us
I20251029 08:52:06.740979 889 tablet_service.cc:1505] Processing CreateTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.741400 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fcf8a3e0f4770a9743fddffa6495f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.742159 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 8991ea78321545be85d5c480841e4751 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.742303 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.742523 757 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.749758 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8991ea78321545be85d5c480841e4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.750195 889 tablet_service.cc:1505] Processing CreateTablet for tablet bc9c221eaa9a4721ba25761206a13645 (DEFAULT_TABLE table=default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.752009 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9c221eaa9a4721ba25761206a13645. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.752815 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 63b08840d0fa40428c61a05b277ad750
I20251029 08:52:06.753516 592 catalog_manager.cc:6719] ReplaceTablet: tablet 63b08840d0fa40428c61a05b277ad750 deleted and replaced by tablet b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:06.754020 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.755019 1438 raft_consensus.cc:493] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.755177 1438 raft_consensus.cc:515] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.755501 890 tablet_service.cc:1505] Processing CreateTablet for tablet 5385fef404a4463595074f0c8ff822d7 (DEFAULT_TABLE table=default.Table_6e61b44def8f4ea283b27494c59f9926 [id=2cfb2b85354d43e7a203d35a4448346d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.755836 1438 leader_election.cc:290] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.755914 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5385fef404a4463595074f0c8ff822d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.756517 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72f64830e924fa48e2fa4cf81ccbba3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.756728 782 raft_consensus.cc:2468] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:06.757148 998 leader_election.cc:304] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:06.757148 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72f64830e924fa48e2fa4cf81ccbba3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.757354 923 raft_consensus.cc:2468] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:06.757436 1438 raft_consensus.cc:2804] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.757534 1438 raft_consensus.cc:493] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.757606 1438 raft_consensus.cc:3060] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.759555 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:06.760504 1438 raft_consensus.cc:515] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.761487 1438 leader_election.cc:290] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.761562 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72f64830e924fa48e2fa4cf81ccbba3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.761740 782 raft_consensus.cc:3060] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.762178 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.764185 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72f64830e924fa48e2fa4cf81ccbba3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:06.764418 923 raft_consensus.cc:3060] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.764981 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d67ccc3d92a84bb8abec8560b542ca91" } new_table_name: "default.Table_70d88123dfe248b6aeb0e6fa51fafc49" modify_external_catalogs: true
I20251029 08:52:06.766685 757 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.767014 1179 ts_tablet_manager.cc:1929] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.767127 1179 log.cc:1199] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e899e183e255498b83a6d740af98de42
I20251029 08:52:06.767290 923 raft_consensus.cc:2468] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:06.767494 1179 ts_tablet_manager.cc:1950] T e899e183e255498b83a6d740af98de42 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:06.767750 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_54314bbc9f794709b664977bd9f32a93" } new_table_name: "default.Table_18a7eae91e5b4ab595c36f630248860f" modify_external_catalogs: true
I20251029 08:52:06.768502 782 raft_consensus.cc:2468] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:06.769002 998 leader_election.cc:304] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:06.769115 896 tablet_service.cc:1505] Processing CreateTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe (DEFAULT_TABLE table=default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.769289 1438 raft_consensus.cc:2804] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:06.769567 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787d6d7f1870450fa21cb9cba34e72fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.769639 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b702ba0aca804d06837f0aa80bae8669"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.770304 1438 raft_consensus.cc:697] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
W20251029 08:52:06.770334 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b702ba0aca804d06837f0aa80bae8669 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.770593 1438 consensus_queue.cc:237] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.772075 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.772400 902 tablet_service.cc:1505] Processing CreateTablet for tablet fd25eeb89e574297a92d8e978a79d768 (DEFAULT_TABLE table=default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.773465 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd25eeb89e574297a92d8e978a79d768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.775755 1461 raft_consensus.cc:493] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.776000 1461 raft_consensus.cc:515] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.777484 1461 leader_election.cc:290] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.776825 894 tablet_service.cc:1505] Processing CreateTablet for tablet e1e2be77182e4259b76d20cae4e9e478 (DEFAULT_TABLE table=default.Table_cc6f14c92a63433f90c93e8bd5956878 [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.777845 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.778218 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e2be77182e4259b76d20cae4e9e478. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.779785 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d64b1a762f124195b4369c096587881d
I20251029 08:52:06.783113 757 tablet_service.cc:1552] Processing DeleteTablet for tablet d64b1a762f124195b4369c096587881d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.783711 891 tablet_service.cc:1505] Processing CreateTablet for tablet d896959a4c684e1b8b68a36125e751b9 (DEFAULT_TABLE table=default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.784138 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d896959a4c684e1b8b68a36125e751b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:06.785235 720 leader_election.cc:343] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.787197 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet d64b1a762f124195b4369c096587881d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.790417 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ddf59288eaf146a0a467f47f4911d920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.790997 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf59288eaf146a0a467f47f4911d920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.791651 891 tablet_service.cc:1505] Processing CreateTablet for tablet 072cf27b0743488c8166e03b5e61c545 (DEFAULT_TABLE table=default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.792060 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072cf27b0743488c8166e03b5e61c545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.792292 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5bd680f724ec45d7816addcd2a67ce84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.792776 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bd680f724ec45d7816addcd2a67ce84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.793422 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c6b0a03da711417aa4c4577b0e8cf2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.793911 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6b0a03da711417aa4c4577b0e8cf2db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.796423 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:06.797856 757 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.801239 895 tablet_service.cc:1505] Processing CreateTablet for tablet c7024e6eb4124109ad1575d216739fe2 (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.801759 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7024e6eb4124109ad1575d216739fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.803309 1183 tablet_bootstrap.cc:492] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:06.803541 1183 ts_tablet_manager.cc:1403] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.441s user 0.006s sys 0.006s
I20251029 08:52:06.804406 1183 raft_consensus.cc:359] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.805203 1183 raft_consensus.cc:385] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.805356 1183 raft_consensus.cc:740] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.805692 1183 consensus_queue.cc:260] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.806960 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c9a1d5e1f68e4f0e8b78641bbdc98123
I20251029 08:52:06.807451 588 catalog_manager.cc:6719] ReplaceTablet: tablet c9a1d5e1f68e4f0e8b78641bbdc98123 deleted and replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689
I20251029 08:52:06.808360 1179 tablet_replica.cc:333] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:06.808616 1179 raft_consensus.cc:2243] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:06.808871 757 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.810207 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f1f60e2c98f47079dc3996b94e0d4c7" } modify_external_catalogs: true
I20251029 08:52:06.810434 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f1f60e2c98f47079dc3996b94e0d4c7" } modify_external_catalogs: true
I20251029 08:52:06.810289 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e899e183e255498b83a6d740af98de42 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
I20251029 08:52:06.810925 888 tablet_service.cc:1505] Processing CreateTablet for tablet d30183814f5e4f2ab9d71fe493eb801c (DEFAULT_TABLE table=default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.811344 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30183814f5e4f2ab9d71fe493eb801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.812105 1178 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.812283 1178 ts_tablet_manager.cc:1403] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.008s
I20251029 08:52:06.813099 1178 raft_consensus.cc:359] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.813308 1178 raft_consensus.cc:385] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.813612 1178 raft_consensus.cc:740] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.813850 1178 consensus_queue.cc:260] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.814353 1178 ts_tablet_manager.cc:1434] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:06.814678 1178 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.813733 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 6426b5a4f5784b3a97475cb23199f916 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.817412 1179 pending_rounds.cc:70] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:06.817570 1179 pending_rounds.cc:77] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037586075807744 op_type: NO_OP noop_request { }
I20251029 08:52:06.817752 1179 raft_consensus.cc:2889] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:06.817924 1179 raft_consensus.cc:2272] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.821273 1179 ts_tablet_manager.cc:1916] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.821555 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:06.822263 591 catalog_manager.cc:6719] ReplaceTablet: tablet a8c5008d9df9419bb4b6d678c6dc27a6 deleted and replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b
I20251029 08:52:06.826300 888 tablet_service.cc:1505] Processing CreateTablet for tablet 840ff254244f43a18679232279e02d1f (DEFAULT_TABLE table=default.Table_c3267734790547088d7d0aabd176ab02 [id=383565dcefdd479288ddfc261cbced6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.826772 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 840ff254244f43a18679232279e02d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.828980 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.829245 757 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:06.829865 1183 ts_tablet_manager.cc:1434] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251029 08:52:06.831730 897 tablet_service.cc:1505] Processing CreateTablet for tablet 19c515944ecf4228a7cca63cfd3565d8 (DEFAULT_TABLE table=default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.832196 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c515944ecf4228a7cca63cfd3565d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.838064 1366 raft_consensus.cc:493] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.838260 1366 raft_consensus.cc:515] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.838903 1366 leader_election.cc:290] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.840638 1183 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:06.844879 1178 tablet_bootstrap.cc:654] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.846174 1183 tablet_bootstrap.cc:654] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.847163 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.854565 1070 raft_consensus.cc:2468] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.847486 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.849627 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:06.856402 593 master_service.cc:954] ReplaceTablet: received request to replace tablet abc307b14aff4c5e937a5151aab1dfb9 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.858985 720 leader_election.cc:304] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:06.859597 1461 raft_consensus.cc:2804] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.859870 1461 raft_consensus.cc:493] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.860033 1461 raft_consensus.cc:3060] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.864189 1461 raft_consensus.cc:515] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:06.864802 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 04b17cb6f63a4064873d0851f4b563cb (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04b17cb6f63a4064873d0851f4b563cb consensus state has no leader
I20251029 08:52:06.865199 1461 leader_election.cc:290] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.849291 1176 ts_tablet_manager.cc:1929] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.866286 1176 log.cc:1199] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/6b8a4b96d3c748daa0f467768b2b7abd
I20251029 08:52:06.866906 1176 ts_tablet_manager.cc:1950] T 6b8a4b96d3c748daa0f467768b2b7abd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.868378 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 6b8a4b96d3c748daa0f467768b2b7abd (table default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]) successfully deleted
I20251029 08:52:06.868878 1176 tablet_replica.cc:333] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.869158 1176 raft_consensus.cc:2243] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.869408 1176 raft_consensus.cc:2272] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.869561 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:06.869827 1070 raft_consensus.cc:3060] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.870383 1176 ts_tablet_manager.cc:1916] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.870465 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.871086 897 tablet_service.cc:1505] Processing CreateTablet for tablet 8991ea78321545be85d5c480841e4751 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.871600 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8991ea78321545be85d5c480841e4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.872725 1070 raft_consensus.cc:2468] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:06.874464 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ca7b76be24b04d34b006fef05f3df689" } new_table_name: "default.Table_4a022119f31d4460af5cdd68baa8521b" modify_external_catalogs: true
I20251029 08:52:06.876926 1353 consensus_queue.cc:1048] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:06.878293 720 leader_election.cc:304] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:06.878571 720 raft_consensus.cc:2686] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:06.883188 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4f80728a7a04583bada485707beb806 consensus state has no leader
I20251029 08:52:06.883617 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 55842c18dbef4513a61dd9629f2e5a4c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:06.883750 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.884120 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.884549 900 tablet_service.cc:1552] Processing DeleteTablet for tablet d64b1a762f124195b4369c096587881d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.884440 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_549035da52504bfa84a95642767fbe31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.884896 900 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:06.885190 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_549035da52504bfa84a95642767fbe31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.885212 900 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.885516 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.887074 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2ec426aafdb6470a85870622c71f3f64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.887732 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ec426aafdb6470a85870622c71f3f64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.888195 1178 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.888379 1178 ts_tablet_manager.cc:1403] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.004s
I20251029 08:52:06.888916 718 leader_election.cc:304] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:06.889376 718 leader_election.cc:343] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 06eb710809754e7abd00a345e8ae591a
I20251029 08:52:06.889513 718 leader_election.cc:304] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:06.889503 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_25a00ec5d08741a18f645f3d31d05ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.889711 718 raft_consensus.cc:2686] T 06eb710809754e7abd00a345e8ae591a P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:06.890120 1353 raft_consensus.cc:493] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:06.890144 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25a00ec5d08741a18f645f3d31d05ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.890280 1353 raft_consensus.cc:515] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.890379 718 leader_election.cc:304] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
W20251029 08:52:06.890532 718 raft_consensus.cc:2686] T 4e808bff7558472099587a0f95c002ff P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:06.891314 1353 leader_election.cc:290] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.891410 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16a2b37e45e444caa3f5117ae9a25304" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.891551 1461 raft_consensus.cc:2749] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:06.891875 861 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.892266 1178 raft_consensus.cc:359] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.892446 1178 raft_consensus.cc:385] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.892506 1178 raft_consensus.cc:740] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:06.893779 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 94738d48801a4fb4a7ad1f5c4022afe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94738d48801a4fb4a7ad1f5c4022afe1 already in progress: creating tablet
I20251029 08:52:06.894649 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e0b459b366cb402db2e7e8fee2b3e855 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]) successfully deleted
I20251029 08:52:06.896216 1178 consensus_queue.cc:260] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:06.897567 861 consensus_peers.cc:597] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:06.900305 1178 ts_tablet_manager.cc:1434] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251029 08:52:06.901556 1183 tablet_bootstrap.cc:492] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:06.901743 1183 ts_tablet_manager.cc:1403] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.002s
I20251029 08:52:06.902532 1183 raft_consensus.cc:359] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.902776 1183 raft_consensus.cc:385] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.902849 1183 raft_consensus.cc:740] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.903108 1183 consensus_queue.cc:260] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.903330 782 raft_consensus.cc:1275] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:06.903625 1183 ts_tablet_manager.cc:1434] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:06.903946 1183 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:06.904886 1353 consensus_queue.cc:1048] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:06.910262 861 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:06.910452 861 leader_election.cc:304] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:06.910979 1353 raft_consensus.cc:2749] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:06.912670 1178 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.914966 1178 tablet_bootstrap.cc:654] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.916059 1183 tablet_bootstrap.cc:654] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.920420 891 tablet_service.cc:1505] Processing CreateTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e (DEFAULT_TABLE table=default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.920998 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2723cdd513ee4e8ca6773b38f785b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.928965 1177 tablet_bootstrap.cc:654] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.930474 1353 raft_consensus.cc:493] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.930914 1353 raft_consensus.cc:515] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.931169 1178 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:06.931324 1178 ts_tablet_manager.cc:1403] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251029 08:52:06.931648 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fd8dcfa769f84911b4151b2c2fa503e7" } new_table_name: "default.Table_4f1be1b978b7433896bb423e059679b9" modify_external_catalogs: true
I20251029 08:52:06.932184 1178 raft_consensus.cc:359] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.932349 1178 raft_consensus.cc:385] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.932289 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.932436 1178 raft_consensus.cc:740] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:06.932845 861 leader_election.cc:343] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.933234 1461 raft_consensus.cc:493] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.933461 1461 raft_consensus.cc:515] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.934937 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:06.935070 1461 leader_election.cc:290] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:06.935416 720 leader_election.cc:343] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.936223 1176 ts_tablet_manager.cc:1929] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:06.936722 1176 log.cc:1199] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/6426b5a4f5784b3a97475cb23199f916
I20251029 08:52:06.937505 1475 mvcc.cc:204] Tried to move back new op lower bound from 7216037588539908096 to 7216037585450450944. Current Snapshot: MvccSnapshot[applied={T|T < 7216037588539908096}]
I20251029 08:52:06.937676 1176 ts_tablet_manager.cc:1950] T 6426b5a4f5784b3a97475cb23199f916 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:06.938407 1353 leader_election.cc:290] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.939179 1183 tablet_bootstrap.cc:492] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:06.939440 1183 ts_tablet_manager.cc:1403] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20251029 08:52:06.940342 1183 raft_consensus.cc:359] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.940599 1183 raft_consensus.cc:385] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:06.940745 1183 raft_consensus.cc:740] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.941080 1183 consensus_queue.cc:260] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:06.941702 1183 ts_tablet_manager.cc:1434] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:06.942104 1183 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:06.942739 1175 ts_tablet_manager.cc:1929] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:06.947209 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_efc1d3e55082423da1deead7cae760f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:06.947978 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc1d3e55082423da1deead7cae760f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.948741 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 6426b5a4f5784b3a97475cb23199f916 (table default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]) successfully deleted
I20251029 08:52:06.949188 1176 tablet_replica.cc:333] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.949415 1176 raft_consensus.cc:2243] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:06.949882 1176 pending_rounds.cc:70] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:06.950007 1176 pending_rounds.cc:77] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037585018978304 op_type: NO_OP noop_request { }
I20251029 08:52:06.938503 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.950229 1176 raft_consensus.cc:2889] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:06.937078 1178 consensus_queue.cc:260] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.950354 783 raft_consensus.cc:2468] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.950495 1366 raft_consensus.cc:493] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.950654 1366 raft_consensus.cc:515] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.951381 1366 leader_election.cc:290] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:06.952716 1178 ts_tablet_manager.cc:1434] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251029 08:52:06.953111 1178 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:06.955179 1178 tablet_bootstrap.cc:654] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:06.950353 1176 raft_consensus.cc:2272] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.957381 1176 ts_tablet_manager.cc:1916] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.958634 1175 log.cc:1199] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7e339c9ed26a48a2a9fe172201c5cc60
I20251029 08:52:06.959363 1175 ts_tablet_manager.cc:1950] T 7e339c9ed26a48a2a9fe172201c5cc60 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:06.960893 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:06.961576 720 leader_election.cc:343] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:06.951056 861 leader_election.cc:304] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:06.951738 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.952252 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.953639 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.966084 924 raft_consensus.cc:2468] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.966243 925 raft_consensus.cc:2468] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.967137 1175 tablet_replica.cc:333] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:06.967346 1175 raft_consensus.cc:2243] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.967517 1175 raft_consensus.cc:2272] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.967819 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]). Expected version 5 got 0
I20251029 08:52:06.968086 1175 ts_tablet_manager.cc:1916] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.968103 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:06.969218 1176 ts_tablet_manager.cc:1929] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:06.969146 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_317871afd272485a9a1789cf2f56a94f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:06.969342 1176 log.cc:1199] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/cf4934c3842a463e857262ceed37e0bc
I20251029 08:52:06.969717 1176 ts_tablet_manager.cc:1950] T cf4934c3842a463e857262ceed37e0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:06.969838 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_317871afd272485a9a1789cf2f56a94f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:06.970794 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet cf4934c3842a463e857262ceed37e0bc (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
I20251029 08:52:06.972183 1434 raft_consensus.cc:2804] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.972194 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb18d609b7424f67a50733661054820d" } new_table_name: "default.Table_af1eb2fb69584b62b01d7ec5d5711afd" modify_external_catalogs: true
I20251029 08:52:06.972309 1434 raft_consensus.cc:493] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.972427 1434 raft_consensus.cc:3060] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.974506 1176 tablet_replica.cc:333] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:06.975041 1176 raft_consensus.cc:2243] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:06.975493 1176 raft_consensus.cc:2272] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:06.976076 757 tablet_service.cc:1505] Processing CreateTablet for tablet ec54e736a3e64dd7a839252c440f778c (DEFAULT_TABLE table=default.Table_19667d766eb94067b16badc787433f2c [id=f13f5c5b7d7742bf9d4a28f2513ec906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.976738 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec54e736a3e64dd7a839252c440f778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.978276 1434 raft_consensus.cc:515] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.979218 1434 leader_election.cc:290] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.945025 890 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:06.950845 1353 raft_consensus.cc:493] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:06.959719 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6426b5a4f5784b3a97475cb23199f916" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:06.979509 921 raft_consensus.cc:3060] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.980218 763 tablet_service.cc:1505] Processing CreateTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (DEFAULT_TABLE table=default.Table_2e2db23cf2c6466b9faa928d7adac713 [id=994af8859eb748c4a3ffee07d64d1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.980615 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.980647 753 tablet_service.cc:1505] Processing CreateTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (DEFAULT_TABLE table=default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:06.981490 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:06.983032 921 raft_consensus.cc:2468] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:06.960273 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.961810 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:06.983693 1353 raft_consensus.cc:515] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:06.984722 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:06.984946 783 raft_consensus.cc:3060] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:06.988104 923 raft_consensus.cc:2468] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:06.989305 1353 leader_election.cc:290] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:06.989560 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:06.989771 784 raft_consensus.cc:2468] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:06.990240 861 leader_election.cc:304] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:06.996272 783 raft_consensus.cc:2468] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:06.998112 1176 ts_tablet_manager.cc:1916] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:06.998713 861 leader_election.cc:304] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:06.999612 1353 raft_consensus.cc:2804] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:06.999764 1353 raft_consensus.cc:493] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:06.999927 1353 raft_consensus.cc:3060] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.003159 1353 raft_consensus.cc:515] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.004313 1353 leader_election.cc:290] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.004541 1353 raft_consensus.cc:2804] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.004622 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:07.004822 783 raft_consensus.cc:3060] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.004959 1353 raft_consensus.cc:697] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:07.005342 1353 consensus_queue.cc:237] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.007786 718 leader_election.cc:304] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.007833 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7e339c9ed26a48a2a9fe172201c5cc60 (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
I20251029 08:52:07.008154 783 raft_consensus.cc:2468] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:07.008237 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8daa66608cf94913afb94ef661b3c898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8daa66608cf94913afb94ef661b3c898 already in progress: creating tablet
I20251029 08:52:07.008546 861 leader_election.cc:304] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:07.008908 1433 raft_consensus.cc:2804] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.009034 1366 raft_consensus.cc:2764] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251029 08:52:07.009073 1433 raft_consensus.cc:697] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:07.011303 1433 consensus_queue.cc:237] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.011715 757 tablet_service.cc:1505] Processing CreateTablet for tablet 449ffc678b874871a672c3059235de6a (DEFAULT_TABLE table=default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.012173 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ffc678b874871a672c3059235de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.014969 753 tablet_service.cc:1505] Processing CreateTablet for tablet 1da3e7940352460c9b2bb060174829c9 (DEFAULT_TABLE table=default.Table_8642fe3704384eaa803ad55dcdcdf6b6 [id=3300362dc3c7436099416c59660b1e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.015429 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da3e7940352460c9b2bb060174829c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.017603 1177 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.017777 1177 ts_tablet_manager.cc:1403] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.283s user 0.001s sys 0.004s
I20251029 08:52:07.018550 1177 raft_consensus.cc:359] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.018747 1177 raft_consensus.cc:385] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.018824 1177 raft_consensus.cc:740] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:06.962852 1487 tablet.cc:1722] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:07.020438 763 tablet_service.cc:1505] Processing CreateTablet for tablet 084a659932ec4996a6d6b160d67efdd0 (DEFAULT_TABLE table=default.Table_67b2c204ba81497680343eaa3b48b255 [id=d88876b1c5514d78892db64b612182ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.020839 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 084a659932ec4996a6d6b160d67efdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.021742 1488 tablet.cc:1722] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:07.026465 1183 tablet_bootstrap.cc:654] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.026551 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:07.027675 861 leader_election.cc:343] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.028465 1035 tablet_service.cc:1505] Processing CreateTablet for tablet ec54e736a3e64dd7a839252c440f778c (DEFAULT_TABLE table=default.Table_19667d766eb94067b16badc787433f2c [id=f13f5c5b7d7742bf9d4a28f2513ec906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.028910 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec54e736a3e64dd7a839252c440f778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.035049 890 tablet_service.cc:1505] Processing CreateTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (DEFAULT_TABLE table=default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.035465 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.035583 718 leader_election.cc:343] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.035723 718 leader_election.cc:304] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:07.036127 718 leader_election.cc:343] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.036268 718 leader_election.cc:304] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.036306 1461 raft_consensus.cc:2749] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.036545 1461 raft_consensus.cc:2749] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.036661 718 leader_election.cc:304] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.036901 1461 raft_consensus.cc:2764] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251029 08:52:07.039661 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.039852 1038 tablet_service.cc:1505] Processing CreateTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (DEFAULT_TABLE table=default.Table_2e2db23cf2c6466b9faa928d7adac713 [id=994af8859eb748c4a3ffee07d64d1e33]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:07.040247 861 leader_election.cc:343] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.040428 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.042088 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 757e34137e884d5ca5cd88fdd03244d8 consensus state has no leader
I20251029 08:52:07.047930 597 catalog_manager.cc:5649] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:07.048936 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 1 got 0
I20251029 08:52:07.049031 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 1da3e7940352460c9b2bb060174829c9 (DEFAULT_TABLE table=default.Table_8642fe3704384eaa803ad55dcdcdf6b6 [id=3300362dc3c7436099416c59660b1e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.049335 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 5f2dd9d6d0854e03a8298625938065c5 (table default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c]). Expected version 1 got 0
I20251029 08:52:07.049428 902 tablet_service.cc:1505] Processing CreateTablet for tablet ec54e736a3e64dd7a839252c440f778c (DEFAULT_TABLE table=default.Table_19667d766eb94067b16badc787433f2c [id=f13f5c5b7d7742bf9d4a28f2513ec906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.049611 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:07.049839 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec54e736a3e64dd7a839252c440f778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.049991 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 449ffc678b874871a672c3059235de6a (DEFAULT_TABLE table=default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.050257 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ed199b700f4f92858c4edb95217d03" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:07.051890 894 tablet_service.cc:1505] Processing CreateTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (DEFAULT_TABLE table=default.Table_2e2db23cf2c6466b9faa928d7adac713 [id=994af8859eb748c4a3ffee07d64d1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.052314 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.052744 861 leader_election.cc:343] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.056485 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 084a659932ec4996a6d6b160d67efdd0 (DEFAULT_TABLE table=default.Table_67b2c204ba81497680343eaa3b48b255 [id=d88876b1c5514d78892db64b612182ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.056730 897 tablet_service.cc:1505] Processing CreateTablet for tablet 084a659932ec4996a6d6b160d67efdd0 (DEFAULT_TABLE table=default.Table_67b2c204ba81497680343eaa3b48b255 [id=d88876b1c5514d78892db64b612182ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.057073 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 084a659932ec4996a6d6b160d67efdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.062216 893 tablet_service.cc:1505] Processing CreateTablet for tablet 449ffc678b874871a672c3059235de6a (DEFAULT_TABLE table=default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.062661 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ffc678b874871a672c3059235de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.064873 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (DEFAULT_TABLE table=default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.066596 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_32a7c0c8b359467995851e70b6448f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.067230 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32a7c0c8b359467995851e70b6448f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.067708 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da3e7940352460c9b2bb060174829c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.067821 1177 consensus_queue.cc:260] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.068571 1177 ts_tablet_manager.cc:1434] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20251029 08:52:07.068871 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ffc678b874871a672c3059235de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.068890 1177 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.069351 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 084a659932ec4996a6d6b160d67efdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.070693 1177 tablet_bootstrap.cc:654] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.071841 895 tablet_service.cc:1505] Processing CreateTablet for tablet 1da3e7940352460c9b2bb060174829c9 (DEFAULT_TABLE table=default.Table_8642fe3704384eaa803ad55dcdcdf6b6 [id=3300362dc3c7436099416c59660b1e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.072263 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da3e7940352460c9b2bb060174829c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.076954 1433 consensus_queue.cc:1048] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:07.095417 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1de2f068a9d64b31b5211cd9c5791707" } modify_external_catalogs: true
I20251029 08:52:07.095696 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1de2f068a9d64b31b5211cd9c5791707" } modify_external_catalogs: true
I20251029 08:52:07.101217 1177 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.101402 1177 ts_tablet_manager.cc:1403] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20251029 08:52:07.101665 1176 ts_tablet_manager.cc:1929] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.103435 1177 raft_consensus.cc:359] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.104660 1177 raft_consensus.cc:385] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.104776 1177 raft_consensus.cc:740] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.105496 1177 consensus_queue.cc:260] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.106812 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.107751 1177 ts_tablet_manager.cc:1434] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:07.109464 1177 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.107271 1179 ts_tablet_manager.cc:1929] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.105852 1183 tablet_bootstrap.cc:492] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.110069 1183 ts_tablet_manager.cc:1403] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.168s user 0.005s sys 0.003s
I20251029 08:52:07.110903 1183 raft_consensus.cc:359] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.111142 1177 tablet_bootstrap.cc:654] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.111163 1183 raft_consensus.cc:385] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.111320 1183 raft_consensus.cc:740] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.111658 1183 consensus_queue.cc:260] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.112768 1183 ts_tablet_manager.cc:1434] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:07.113137 1183 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.115392 1176 log.cc:1199] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e72f64830e924fa48e2fa4cf81ccbba3
I20251029 08:52:07.115814 1183 tablet_bootstrap.cc:654] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.116690 1179 log.cc:1199] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/6b8a4b96d3c748daa0f467768b2b7abd
I20251029 08:52:07.124513 1179 ts_tablet_manager.cc:1950] T 6b8a4b96d3c748daa0f467768b2b7abd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.130795 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 6b8a4b96d3c748daa0f467768b2b7abd (table default.Table_f067b822946a45ea946eeabd484c2975 [id=a8b0dcd47ac44a98bb728bd976258301]) successfully deleted
I20251029 08:52:07.133536 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9f8030a3d166447fbd35405c27dc35ec" } modify_external_catalogs: true
I20251029 08:52:07.133782 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9f8030a3d166447fbd35405c27dc35ec" } modify_external_catalogs: true
I20251029 08:52:07.135522 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:07.135577 1179 tablet_replica.cc:333] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.135852 1179 raft_consensus.cc:2243] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.136044 1179 raft_consensus.cc:2272] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.136327 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fa76b534de514a8cb76768b632ed61de" } modify_external_catalogs: true
I20251029 08:52:07.136538 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fa76b534de514a8cb76768b632ed61de" } modify_external_catalogs: true
I20251029 08:52:07.136613 1176 ts_tablet_manager.cc:1950] T e72f64830e924fa48e2fa4cf81ccbba3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.136644 1179 ts_tablet_manager.cc:1916] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.138105 1176 tablet_replica.cc:333] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.138332 1176 raft_consensus.cc:2243] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.138490 1176 raft_consensus.cc:2272] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.139027 1176 ts_tablet_manager.cc:1916] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.139321 1461 raft_consensus.cc:493] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.139535 1461 raft_consensus.cc:515] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.140365 1461 leader_election.cc:290] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.141414 1178 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.141578 1178 ts_tablet_manager.cc:1403] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.189s user 0.013s sys 0.007s
I20251029 08:52:07.141711 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.143146 1178 raft_consensus.cc:359] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.143447 1178 raft_consensus.cc:385] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:07.143505 720 leader_election.cc:343] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.144215 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.144299 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e72f64830e924fa48e2fa4cf81ccbba3 (table default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]) successfully deleted
I20251029 08:52:07.144939 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.147593 1438 raft_consensus.cc:493] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.147787 1438 raft_consensus.cc:515] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.148463 1438 leader_election.cc:290] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.149009 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:07.149012 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d64b1a762f124195b4369c096587881d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.149283 782 raft_consensus.cc:2393] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:07.149252 1487 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:07.149649 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.150403 996 leader_election.cc:304] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.150686 1438 raft_consensus.cc:3060] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.153052 1177 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.153293 1177 ts_tablet_manager.cc:1403] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.015s
I20251029 08:52:07.153630 1438 raft_consensus.cc:2749] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.143537 1178 raft_consensus.cc:740] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.154132 1178 consensus_queue.cc:260] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.154557 1460 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:07.154685 1178 ts_tablet_manager.cc:1434] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 08:52:07.155494 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.156466 1177 raft_consensus.cc:359] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.157080 1177 raft_consensus.cc:385] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.157150 1177 raft_consensus.cc:740] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.158771 1177 consensus_queue.cc:260] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.160634 1177 ts_tablet_manager.cc:1434] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 08:52:07.168512 1183 tablet_bootstrap.cc:492] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.168731 1183 ts_tablet_manager.cc:1403] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20251029 08:52:07.169665 1183 raft_consensus.cc:359] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.169916 1183 raft_consensus.cc:385] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.170045 1183 raft_consensus.cc:740] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.170313 1183 consensus_queue.cc:260] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.171571 1176 ts_tablet_manager.cc:1929] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.172428 1176 log.cc:1199] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/24ed199b700f4f92858c4edb95217d03
I20251029 08:52:07.173085 1176 ts_tablet_manager.cc:1950] T 24ed199b700f4f92858c4edb95217d03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.172067 1183 ts_tablet_manager.cc:1434] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:07.174672 1183 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:07.175890 861 consensus_peers.cc:597] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13eeb2e8951b421e9d6f8f228a956e3d. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:07.176334 718 leader_election.cc:343] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.176469 1177 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.176486 718 leader_election.cc:304] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.176900 1366 raft_consensus.cc:2749] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.177172 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 24ed199b700f4f92858c4edb95217d03 (table default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]) successfully deleted
I20251029 08:52:07.178192 593 master_service.cc:954] ReplaceTablet: received request to replace tablet cf4934c3842a463e857262ceed37e0bc from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.178833 1177 tablet_bootstrap.cc:654] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.183796 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192" } modify_external_catalogs: true
I20251029 08:52:07.184784 1488 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:07.184772 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192" } modify_external_catalogs: true
I20251029 08:52:07.189975 1183 tablet_bootstrap.cc:654] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.191983 1178 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.194494 1178 tablet_bootstrap.cc:654] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.196993 1179 ts_tablet_manager.cc:1929] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:07.198299 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 351047c725f8488abbd074bd40790b48 has no consensus state
I20251029 08:52:07.201998 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_601d7697cc9b40c0ac3fa0c5ae003d8c" } new_table_name: "default.Table_cd7b287bef4d4e1da804f48f31500707" modify_external_catalogs: true
I20251029 08:52:07.204486 1179 log.cc:1199] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/db9cf2a8f2254a38a725bcd3c83e8509
I20251029 08:52:07.204980 1179 ts_tablet_manager.cc:1950] T db9cf2a8f2254a38a725bcd3c83e8509 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.207522 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet db9cf2a8f2254a38a725bcd3c83e8509 (table default.Table_cd11e136b8054bdfb9c12527b0fc5777 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
I20251029 08:52:07.208269 1179 tablet_replica.cc:333] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.208660 1179 raft_consensus.cc:2243] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.209193 1179 raft_consensus.cc:2272] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.210282 1179 ts_tablet_manager.cc:1916] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.210995 1183 tablet_bootstrap.cc:492] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.211238 1183 ts_tablet_manager.cc:1403] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20251029 08:52:07.212070 1183 raft_consensus.cc:359] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.212321 1183 raft_consensus.cc:385] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.212431 1183 raft_consensus.cc:740] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.212747 1183 consensus_queue.cc:260] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.214811 1183 ts_tablet_manager.cc:1434] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:07.215314 1183 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.216367 1353 raft_consensus.cc:493] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.216581 1353 raft_consensus.cc:515] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.217454 1175 ts_tablet_manager.cc:1929] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.217639 1175 log.cc:1199] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:07.218127 1175 ts_tablet_manager.cc:1950] T 16a2b37e45e444caa3f5117ae9a25304 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.218192 1353 leader_election.cc:290] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.218976 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:07.219521 861 leader_election.cc:343] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.219797 1175 tablet_replica.cc:333] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.220139 1175 raft_consensus.cc:2243] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.220368 1175 raft_consensus.cc:2272] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.220415 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_fd8dcfa769f84911b4151b2c2fa503e7 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 1 got 0
I20251029 08:52:07.221149 1175 ts_tablet_manager.cc:1916] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.224814 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 16a2b37e45e444caa3f5117ae9a25304 (table default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]) successfully deleted
I20251029 08:52:07.224980 1183 tablet_bootstrap.cc:654] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.225119 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 39dc5ebeb36043e0aa5955bacd06c2ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39dc5ebeb36043e0aa5955bacd06c2ef already in progress: creating tablet
W20251029 08:52:07.225308 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2c706343d44a4807b1ea60c1b1e1ea49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c706343d44a4807b1ea60c1b1e1ea49 already in progress: creating tablet
I20251029 08:52:07.226576 902 tablet_service.cc:1552] Processing DeleteTablet for tablet e72f64830e924fa48e2fa4cf81ccbba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.227804 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:07.228060 782 raft_consensus.cc:2468] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:07.229115 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b78810487e57403d9cc5c7026930e2f2" } new_table_name: "default.Table_81ab02d14c514f1f9c7622f3776b4552" modify_external_catalogs: true
I20251029 08:52:07.230562 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_770bebdc34fe40eab00f06e1d53a4d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.231156 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_770bebdc34fe40eab00f06e1d53a4d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.231534 861 leader_election.cc:304] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.232203 1353 raft_consensus.cc:2804] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.232388 1353 raft_consensus.cc:493] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.232553 1353 raft_consensus.cc:3060] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.233108 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_433d1398ff1f4ba88a75224fe2e4d069"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.233655 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_433d1398ff1f4ba88a75224fe2e4d069 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.233623 1487 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:07.234423 1460 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:07.234766 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_208a79319e20472782aba43b998349ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.235250 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_208a79319e20472782aba43b998349ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.236806 1353 raft_consensus.cc:515] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.237762 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:07.238976 1175 ts_tablet_manager.cc:1929] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.238936 1488 tablet.cc:1722] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:07.239210 1175 log.cc:1199] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/cc1d1d5b2a9e4051bbcd25be1578ee73
W20251029 08:52:07.239522 861 leader_election.cc:343] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.240132 1353 leader_election.cc:290] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.240720 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9cf61d6137544019bca25acf2431986" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:07.240934 782 raft_consensus.cc:3060] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.241547 1175 ts_tablet_manager.cc:1950] T cc1d1d5b2a9e4051bbcd25be1578ee73 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:07.242394 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 57e2b19a227740a28ae601d77e62597b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e2b19a227740a28ae601d77e62597b already in progress: creating tablet
I20251029 08:52:07.242551 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet cc1d1d5b2a9e4051bbcd25be1578ee73 (table default.Table_75195faaf53b406c889072cd3a86bba0 [id=f06a098030a14576b20d373c3adaf8b7]) successfully deleted
I20251029 08:52:07.242916 1175 tablet_replica.cc:333] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.243094 1175 raft_consensus.cc:2243] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.243257 1175 raft_consensus.cc:2272] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:07.243656 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet abc307b14aff4c5e937a5151aab1dfb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc307b14aff4c5e937a5151aab1dfb9 already in progress: creating tablet
I20251029 08:52:07.243883 1175 ts_tablet_manager.cc:1916] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:07.248458 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 63b08840d0fa40428c61a05b277ad750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63b08840d0fa40428c61a05b277ad750 already in progress: creating tablet
W20251029 08:52:07.249255 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a20f26d0f7874ccb960ffe4d27e854e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20f26d0f7874ccb960ffe4d27e854e1 already in progress: creating tablet
I20251029 08:52:07.249667 1176 tablet_replica.cc:333] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.249913 1176 raft_consensus.cc:2243] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.250131 1176 raft_consensus.cc:2272] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.250705 1176 ts_tablet_manager.cc:1916] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.253515 1179 ts_tablet_manager.cc:1929] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.255911 1178 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.256127 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:07.256173 1178 ts_tablet_manager.cc:1403] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.004s
I20251029 08:52:07.257100 897 tablet_service.cc:1552] Processing DeleteTablet for tablet a20b64ff177241309e5d59e84f0bafa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.257521 1178 raft_consensus.cc:359] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.257705 1178 raft_consensus.cc:385] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.257786 1178 raft_consensus.cc:740] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.258033 1178 consensus_queue.cc:260] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.258313 1175 ts_tablet_manager.cc:1929] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.258445 1175 log.cc:1199] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/b4ceeaba655542dab2556d3988b6e841
I20251029 08:52:07.258548 1178 ts_tablet_manager.cc:1434] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:07.258922 1175 ts_tablet_manager.cc:1950] T b4ceeaba655542dab2556d3988b6e841 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.259959 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a20b64ff177241309e5d59e84f0bafa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.260074 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet b4ceeaba655542dab2556d3988b6e841 (table default.Table_1c778ace3a094ba7a37d7def4c6497a7 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:07.260452 1175 tablet_replica.cc:333] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.260705 1175 raft_consensus.cc:2243] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.260897 1175 raft_consensus.cc:2272] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.261472 1175 ts_tablet_manager.cc:1916] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.244624 782 raft_consensus.cc:2468] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:07.263818 861 leader_election.cc:304] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.264317 1353 raft_consensus.cc:2804] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.264881 1353 raft_consensus.cc:697] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:07.265261 1353 consensus_queue.cc:237] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.268946 588 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ca962f295ebc4aa0a5c4efbf944b8b11
I20251029 08:52:07.259569 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a20b64ff177241309e5d59e84f0bafa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.269075 1183 tablet_bootstrap.cc:492] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.269487 1178 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.269942 1183 ts_tablet_manager.cc:1403] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.005s
I20251029 08:52:07.270843 1183 raft_consensus.cc:359] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.271118 1183 raft_consensus.cc:385] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.271253 1183 raft_consensus.cc:740] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.271569 1183 consensus_queue.cc:260] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.271863 1178 tablet_bootstrap.cc:654] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.275679 1183 ts_tablet_manager.cc:1434] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 08:52:07.275683 1175 ts_tablet_manager.cc:1929] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.275820 1175 log.cc:1199] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/453911c5a29f43e7a21fa2c47ef264e8
I20251029 08:52:07.276659 1175 ts_tablet_manager.cc:1950] T 453911c5a29f43e7a21fa2c47ef264e8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.277729 1183 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.277838 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 453911c5a29f43e7a21fa2c47ef264e8 (table default.Table_bfa1f140404f4768b6ae7c7fabc75d26 [id=45e9519b40cf49c3be6472b1025c5e18]) successfully deleted
I20251029 08:52:07.278281 1448 raft_consensus.cc:493] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.278446 1448 raft_consensus.cc:515] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.279074 1448 leader_election.cc:290] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.279407 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b273c488f33c498890f1739380b18239" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.277880 1179 log.cc:1199] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:07.279886 1175 tablet_replica.cc:333] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.280045 1177 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
W20251029 08:52:07.279990 861 leader_election.cc:343] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.280180 1177 ts_tablet_manager.cc:1403] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.003s
I20251029 08:52:07.280342 1175 raft_consensus.cc:2243] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:07.280645 1175 pending_rounds.cc:70] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:07.280766 1175 pending_rounds.cc:77] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037589041070080 op_type: NO_OP noop_request { }
I20251029 08:52:07.280961 1175 raft_consensus.cc:2889] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:07.281121 1175 raft_consensus.cc:2272] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.281169 1177 raft_consensus.cc:359] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.281316 1177 raft_consensus.cc:385] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.281404 1177 raft_consensus.cc:740] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.281504 1438 raft_consensus.cc:493] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.281672 1438 raft_consensus.cc:515] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.281872 1177 consensus_queue.cc:260] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.282459 1438 leader_election.cc:290] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.282979 1179 ts_tablet_manager.cc:1950] T 13eeb2e8951b421e9d6f8f228a956e3d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:07.283337 996 leader_election.cc:343] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:07.283411 1178 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.282095 1175 ts_tablet_manager.cc:1916] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.283589 1178 ts_tablet_manager.cc:1403] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20251029 08:52:07.284065 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 13eeb2e8951b421e9d6f8f228a956e3d (table default.Table_2e15dbeda3ab4c4b8473ccd12597b270 [id=407839d19657473883ad497873b78fb2]) successfully deleted
I20251029 08:52:07.284250 1177 ts_tablet_manager.cc:1434] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251029 08:52:07.284377 998 leader_election.cc:343] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:07.284415 1178 raft_consensus.cc:359] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.284497 998 leader_election.cc:304] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.284559 1177 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.284592 1178 raft_consensus.cc:385] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.284658 1178 raft_consensus.cc:740] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.285056 1438 raft_consensus.cc:2749] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.286368 1177 tablet_bootstrap.cc:654] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.287209 1183 tablet_bootstrap.cc:654] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.287535 861 leader_election.cc:343] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b273c488f33c498890f1739380b18239
I20251029 08:52:07.287683 861 leader_election.cc:304] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:07.287883 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 16a2b37e45e444caa3f5117ae9a25304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a2b37e45e444caa3f5117ae9a25304 already in progress: creating tablet
I20251029 08:52:07.288061 1448 raft_consensus.cc:2749] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.291353 1178 consensus_queue.cc:260] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:07.292064 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 39dc5ebeb36043e0aa5955bacd06c2ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39dc5ebeb36043e0aa5955bacd06c2ef already in progress: creating tablet
I20251029 08:52:07.291519 1438 raft_consensus.cc:493] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.292366 1438 raft_consensus.cc:515] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.292445 1178 ts_tablet_manager.cc:1434] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251029 08:52:07.292747 1178 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.292770 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ef0a7b31ced24388bc9d9964a42fa203
I20251029 08:52:07.292964 1438 leader_election.cc:290] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:07.293567 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d22c6f8438dc46d79b03e79db3d79cac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:07.293782 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "904523f7164849139cddb1100e95b824" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.293965 923 raft_consensus.cc:2468] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:07.294133 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d22c6f8438dc46d79b03e79db3d79cac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.294142 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.294368 753 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.294499 996 leader_election.cc:304] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:07.294760 1438 raft_consensus.cc:2804] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.294842 1438 raft_consensus.cc:493] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.294904 1438 raft_consensus.cc:3060] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.296801 897 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.297843 1438 raft_consensus.cc:515] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.298557 1178 tablet_bootstrap.cc:654] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.299074 1438 leader_election.cc:290] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:07.299161 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "904523f7164849139cddb1100e95b824" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:07.299340 923 raft_consensus.cc:3060] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:07.300622 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2c706343d44a4807b1ea60c1b1e1ea49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c706343d44a4807b1ea60c1b1e1ea49 already in progress: creating tablet
W20251029 08:52:07.301091 998 leader_election.cc:343] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 904523f7164849139cddb1100e95b824
I20251029 08:52:07.301914 1175 ts_tablet_manager.cc:1929] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.302090 1175 log.cc:1199] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/24ed199b700f4f92858c4edb95217d03
I20251029 08:52:07.302443 923 raft_consensus.cc:2468] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:07.302906 996 leader_election.cc:304] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:07.303012 1175 ts_tablet_manager.cc:1950] T 24ed199b700f4f92858c4edb95217d03 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.303390 1438 raft_consensus.cc:2804] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.304066 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 24ed199b700f4f92858c4edb95217d03 (table default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]) successfully deleted
W20251029 08:52:07.304344 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 57e2b19a227740a28ae601d77e62597b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e2b19a227740a28ae601d77e62597b already in progress: creating tablet
I20251029 08:52:07.304483 1175 tablet_replica.cc:333] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.304687 1175 raft_consensus.cc:2243] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:07.304724 998 leader_election.cc:343] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 904523f7164849139cddb1100e95b824
I20251029 08:52:07.304929 1175 raft_consensus.cc:2272] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.305357 1179 tablet_replica.cc:333] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.305968 1179 raft_consensus.cc:2243] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:07.306653 1179 pending_rounds.cc:70] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:07.307214 1179 pending_rounds.cc:77] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037588055699456 op_type: NO_OP noop_request { }
I20251029 08:52:07.307670 1179 raft_consensus.cc:2889] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:07.308053 1179 raft_consensus.cc:2272] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.309317 1179 ts_tablet_manager.cc:1916] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:07.309527 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc57cd5a8d3c42ba90f2011084e8b4bd consensus state has no leader
I20251029 08:52:07.309845 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a7a673ba4fd4abcbfc3493c38dcc257 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.310956 1177 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.311569 1177 ts_tablet_manager.cc:1403] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20251029 08:52:07.313127 1177 raft_consensus.cc:359] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.314808 1177 raft_consensus.cc:385] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.315064 1177 raft_consensus.cc:740] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.315783 1177 consensus_queue.cc:260] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.317301 1176 ts_tablet_manager.cc:1929] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.317749 1177 ts_tablet_manager.cc:1434] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:07.319696 1183 tablet_bootstrap.cc:492] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.319902 1177 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.319041 1178 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.322021 1178 ts_tablet_manager.cc:1403] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20251029 08:52:07.322171 1176 log.cc:1199] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d64b1a762f124195b4369c096587881d
I20251029 08:52:07.320287 1438 raft_consensus.cc:697] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:07.322583 1438 consensus_queue.cc:237] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.323698 1183 ts_tablet_manager.cc:1403] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
W20251029 08:52:07.323992 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add0605edf5c4511ae4c8d9cb7f0babb consensus state has no leader
I20251029 08:52:07.322878 1178 raft_consensus.cc:359] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.324151 1178 raft_consensus.cc:385] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.324267 1178 raft_consensus.cc:740] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.324312 589 master_service.cc:954] ReplaceTablet: received request to replace tablet b273c488f33c498890f1739380b18239 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.324590 1176 ts_tablet_manager.cc:1950] T d64b1a762f124195b4369c096587881d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.324522 1178 consensus_queue.cc:260] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.324978 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 363fcf8a3e0f4770a9743fddffa6495f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.325116 1366 raft_consensus.cc:493] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.325246 1366 raft_consensus.cc:515] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.325374 1178 ts_tablet_manager.cc:1434] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:07.325670 1178 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.325928 1366 leader_election.cc:290] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:07.326550 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a73a891fb6034f989d29a3f837303d84" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.326761 923 raft_consensus.cc:2468] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:07.327064 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d64b1a762f124195b4369c096587881d (table default.Table_8325dc8d37f147e79a2672a42a930de9 [id=12d36151f38e4a5783f82884f1eea461]) successfully deleted
I20251029 08:52:07.327606 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a73a891fb6034f989d29a3f837303d84" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.327908 1178 tablet_bootstrap.cc:654] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.328353 718 leader_election.cc:304] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:07.328532 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 23b534aaa51b4fad9dc3fc915de947f1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.328671 1366 raft_consensus.cc:2804] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.328790 1366 raft_consensus.cc:493] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.328881 1366 raft_consensus.cc:3060] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:07.329066 720 leader_election.cc:343] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.332521 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e6dc65fd6a3f4d53a1c27c9fecf32490" } new_table_name: "default.Table_4c3334f5e9ce4296866318c8b7fbe1eb" modify_external_catalogs: true
I20251029 08:52:07.333979 1366 raft_consensus.cc:515] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.335238 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a2841227adaf4c67aed78c15574f5188"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:07.335666 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a73a891fb6034f989d29a3f837303d84" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:07.335871 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2841227adaf4c67aed78c15574f5188 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:07.336189 720 leader_election.cc:343] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.336884 1366 leader_election.cc:290] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:07.337213 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a73a891fb6034f989d29a3f837303d84" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:07.337436 923 raft_consensus.cc:3060] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.337585 1183 raft_consensus.cc:359] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.337803 1183 raft_consensus.cc:385] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.337877 1183 raft_consensus.cc:740] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:07.339265 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e626994e96d54c0d913fdedc85fd1155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e626994e96d54c0d913fdedc85fd1155 already in progress: creating tablet
W20251029 08:52:07.340209 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a1d5e1f68e4f0e8b78641bbdc98123 already in progress: creating tablet
I20251029 08:52:07.338519 1183 consensus_queue.cc:260] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.340965 1183 ts_tablet_manager.cc:1434] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
W20251029 08:52:07.342676 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a8c5008d9df9419bb4b6d678c6dc27a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c5008d9df9419bb4b6d678c6dc27a6 already in progress: creating tablet
I20251029 08:52:07.343495 923 raft_consensus.cc:2468] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:07.343698 1177 tablet_bootstrap.cc:654] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.343729 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc already in progress: creating tablet
I20251029 08:52:07.344250 718 leader_election.cc:304] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.344309 1175 ts_tablet_manager.cc:1916] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.344782 1366 raft_consensus.cc:2804] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.345381 1366 raft_consensus.cc:697] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:07.345781 1366 consensus_queue.cc:237] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.346619 1178 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.346784 1178 ts_tablet_manager.cc:1403] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20251029 08:52:07.348553 1178 raft_consensus.cc:359] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.348825 1178 raft_consensus.cc:385] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.348929 1178 raft_consensus.cc:740] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.349217 1178 consensus_queue.cc:260] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.350878 1179 ts_tablet_manager.cc:1929] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.352877 1178 ts_tablet_manager.cc:1434] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251029 08:52:07.353751 1177 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.353914 1177 ts_tablet_manager.cc:1403] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20251029 08:52:07.358690 1178 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.358880 1179 log.cc:1199] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e72f64830e924fa48e2fa4cf81ccbba3
I20251029 08:52:07.359125 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7c5175dfc80a4536bb7ff8968998073a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:07.359414 1179 ts_tablet_manager.cc:1950] T e72f64830e924fa48e2fa4cf81ccbba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:07.360235 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c5175dfc80a4536bb7ff8968998073a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.360515 1178 tablet_bootstrap.cc:654] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.361548 1183 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.362637 1176 tablet_replica.cc:333] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.363107 1176 raft_consensus.cc:2243] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.363090 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_7b45e52828bf4727815bfbb7c1ddb9c3 [id=63588e39919a481c856255fd848bbf98]). Expected version 1 got 0
I20251029 08:52:07.363348 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:07.363576 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 2 got 1
I20251029 08:52:07.363620 1176 raft_consensus.cc:2272] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.366066 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 2 got 1
I20251029 08:52:07.366322 592 catalog_manager.cc:5649] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:07.366743 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:07.371963 897 tablet_service.cc:1505] Processing CreateTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.372412 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f40aafe7dc94851a67abd84d80d7d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.372637 1175 ts_tablet_manager.cc:1929] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.372747 1175 log.cc:1199] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/cf4934c3842a463e857262ceed37e0bc
I20251029 08:52:07.373162 1175 ts_tablet_manager.cc:1950] T cf4934c3842a463e857262ceed37e0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.374837 1177 raft_consensus.cc:359] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.375032 1177 raft_consensus.cc:385] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.375111 1177 raft_consensus.cc:740] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.376328 902 tablet_service.cc:1505] Processing CreateTablet for tablet 772de9216f2647f8ae52f594df272427 (DEFAULT_TABLE table=default.Table_c6b0a03da711417aa4c4577b0e8cf2db [id=9089031d844641f0b238f724fdbd2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.376703 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772de9216f2647f8ae52f594df272427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.377986 1176 ts_tablet_manager.cc:1916] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.380031 1183 tablet_bootstrap.cc:654] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.380285 1179 tablet_replica.cc:333] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.380295 1353 raft_consensus.cc:493] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.380461 1353 raft_consensus.cc:515] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.381135 1353 leader_election.cc:290] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.381508 1178 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.381666 1178 ts_tablet_manager.cc:1403] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251029 08:52:07.382485 1178 raft_consensus.cc:359] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.382676 1178 raft_consensus.cc:385] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.382774 1178 raft_consensus.cc:740] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.382987 1179 raft_consensus.cc:2243] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.383030 1178 consensus_queue.cc:260] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.383196 1179 raft_consensus.cc:2272] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.383162 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb6b86f39c994ea5930a6a970830093f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.383605 1178 ts_tablet_manager.cc:1434] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 08:52:07.383937 1178 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.383930 1179 ts_tablet_manager.cc:1916] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:07.384843 861 leader_election.cc:343] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.386297 1178 tablet_bootstrap.cc:654] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.388741 861 leader_election.cc:343] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bb6b86f39c994ea5930a6a970830093f
I20251029 08:52:07.388881 861 leader_election.cc:304] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:07.389077 1175 tablet_replica.cc:333] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.389282 1175 raft_consensus.cc:2243] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.389451 1175 raft_consensus.cc:2272] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.389506 1353 raft_consensus.cc:2749] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.390213 1177 consensus_queue.cc:260] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.391211 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e72f64830e924fa48e2fa4cf81ccbba3 (table default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]) successfully deleted
W20251029 08:52:07.391532 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet abc307b14aff4c5e937a5151aab1dfb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc307b14aff4c5e937a5151aab1dfb9 already in progress: creating tablet
W20251029 08:52:07.391750 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 63b08840d0fa40428c61a05b277ad750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63b08840d0fa40428c61a05b277ad750 already in progress: creating tablet
I20251029 08:52:07.391819 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet cf4934c3842a463e857262ceed37e0bc (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
W20251029 08:52:07.391990 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a20f26d0f7874ccb960ffe4d27e854e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20f26d0f7874ccb960ffe4d27e854e1 already in progress: creating tablet
I20251029 08:52:07.392876 763 tablet_service.cc:1505] Processing CreateTablet for tablet 772de9216f2647f8ae52f594df272427 (DEFAULT_TABLE table=default.Table_c6b0a03da711417aa4c4577b0e8cf2db [id=9089031d844641f0b238f724fdbd2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.392953 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.393324 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772de9216f2647f8ae52f594df272427. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.393824 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f33c0bbf104046ccb86ff313f08da22a consensus state has no leader
I20251029 08:52:07.394685 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f40aafe7dc94851a67abd84d80d7d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.394924 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_10573b6a1c074383b5808291ca1505ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.395491 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10573b6a1c074383b5808291ca1505ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.396271 757 tablet_service.cc:1505] Processing CreateTablet for tablet bda3e143699b41b6bba308c83bbb3e0d (DEFAULT_TABLE table=default.Table_5bd680f724ec45d7816addcd2a67ce84 [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.396255 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 772de9216f2647f8ae52f594df272427 (DEFAULT_TABLE table=default.Table_c6b0a03da711417aa4c4577b0e8cf2db [id=9089031d844641f0b238f724fdbd2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.396665 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772de9216f2647f8ae52f594df272427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.396709 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda3e143699b41b6bba308c83bbb3e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.397557 1177 ts_tablet_manager.cc:1434] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20251029 08:52:07.398079 1175 ts_tablet_manager.cc:1916] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.395933 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.400409 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f40aafe7dc94851a67abd84d80d7d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.404731 1177 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.395929 1042 tablet_service.cc:1505] Processing CreateTablet for tablet bda3e143699b41b6bba308c83bbb3e0d (DEFAULT_TABLE table=default.Table_5bd680f724ec45d7816addcd2a67ce84 [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.405179 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda3e143699b41b6bba308c83bbb3e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.406459 1177 tablet_bootstrap.cc:654] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.406554 861 consensus_peers.cc:597] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:07.407544 891 tablet_service.cc:1505] Processing CreateTablet for tablet bda3e143699b41b6bba308c83bbb3e0d (DEFAULT_TABLE table=default.Table_5bd680f724ec45d7816addcd2a67ce84 [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.407922 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda3e143699b41b6bba308c83bbb3e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.411720 895 tablet_service.cc:1505] Processing CreateTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.412230 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2bb1075c4418eb10f4493f5e73c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.423027 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a483eeed2297411daf8f8908b370db56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.423664 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a483eeed2297411daf8f8908b370db56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.423988 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:07.424474 597 catalog_manager.cc:6719] ReplaceTablet: tablet 23b534aaa51b4fad9dc3fc915de947f1 deleted and replaced by tablet fc04616301354cfe873c825a9f5d7ab6
I20251029 08:52:07.424824 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_390ea1069fb74e8fa400c7c309ec231e" } modify_external_catalogs: true
I20251029 08:52:07.424993 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_390ea1069fb74e8fa400c7c309ec231e" } modify_external_catalogs: true
I20251029 08:52:07.425066 1176 ts_tablet_manager.cc:1929] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.425212 1176 log.cc:1199] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:07.425662 1176 ts_tablet_manager.cc:1950] T a20b64ff177241309e5d59e84f0bafa2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.426491 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:07.426927 595 catalog_manager.cc:6719] ReplaceTablet: tablet 363fcf8a3e0f4770a9743fddffa6495f deleted and replaced by tablet bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:07.427292 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_565b58db3efc46a9bd80cb9ae264bab1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.427784 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_565b58db3efc46a9bd80cb9ae264bab1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.428297 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0ff4e625abf14fbd94a8102dfa5fc993
I20251029 08:52:07.428567 1177 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.428766 1177 ts_tablet_manager.cc:1403] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20251029 08:52:07.429080 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b89cba8e4a642dabf87ae4f9a509895" } new_table_name: "default.Table_897033c1be1f463fb7ef32a818c87de3" modify_external_catalogs: true
I20251029 08:52:07.429677 1177 raft_consensus.cc:359] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.430073 1177 raft_consensus.cc:385] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.430197 1177 raft_consensus.cc:740] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.430635 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.430891 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.431109 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.431393 1177 consensus_queue.cc:260] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.433975 1177 ts_tablet_manager.cc:1434] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:07.432608 1179 ts_tablet_manager.cc:1929] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.435137 1179 log.cc:1199] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d64b1a762f124195b4369c096587881d
I20251029 08:52:07.435742 1179 ts_tablet_manager.cc:1950] T d64b1a762f124195b4369c096587881d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.436542 1177 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.437162 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]). Expected version 2 got 1
I20251029 08:52:07.438508 1179 tablet_replica.cc:333] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.438977 1177 tablet_bootstrap.cc:654] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.438942 1179 raft_consensus.cc:2243] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.439155 588 master_service.cc:954] ReplaceTablet: received request to replace tablet f33c0bbf104046ccb86ff313f08da22a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.439744 1183 tablet_bootstrap.cc:492] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.440042 1183 ts_tablet_manager.cc:1403] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.002s
W20251029 08:52:07.441740 861 consensus_peers.cc:597] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64b1a762f124195b4369c096587881d. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:07.442200 1183 raft_consensus.cc:359] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.442478 1183 raft_consensus.cc:385] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.442611 1183 raft_consensus.cc:740] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.442914 1183 consensus_queue.cc:260] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.443167 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c] alter complete (version 1)
I20251029 08:52:07.443954 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8a32e8fc09e448c1af180a8d85d9158d" } new_table_name: "default.Table_acdad95817bc4aaa86670404d517c37d" modify_external_catalogs: true
I20251029 08:52:07.448458 1183 ts_tablet_manager.cc:1434] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:07.448751 1177 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.448940 1177 ts_tablet_manager.cc:1403] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20251029 08:52:07.449554 1183 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.449771 1177 raft_consensus.cc:359] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.450011 1177 raft_consensus.cc:385] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.450115 1177 raft_consensus.cc:740] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.450400 1177 consensus_queue.cc:260] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.450743 1179 raft_consensus.cc:2272] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.450584 592 catalog_manager.cc:5649] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:07.451295 1179 ts_tablet_manager.cc:1916] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.452400 1353 raft_consensus.cc:493] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.452574 1353 raft_consensus.cc:515] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.453224 1353 leader_election.cc:290] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.453424 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.453510 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.453696 894 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.454170 889 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.455523 763 tablet_service.cc:1505] Processing CreateTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.455936 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2bb1075c4418eb10f4493f5e73c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.456158 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.456240 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.456388 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.456580 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.456766 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.457536 1461 raft_consensus.cc:493] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.457700 1461 raft_consensus.cc:515] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.457834 1176 tablet_replica.cc:333] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.458005 1176 raft_consensus.cc:2243] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.458191 1176 raft_consensus.cc:2272] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.458331 1461 leader_election.cc:290] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.458740 1176 ts_tablet_manager.cc:1916] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.459640 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4cdf03acb92449d847aec897868b967" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
W20251029 08:52:07.461364 861 leader_election.cc:343] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: Tablet not RUNNING: STOPPED
I20251029 08:52:07.455147 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.455178 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 06c08679f1e746538bfa2c80d4fbdba3 with delete_type TABLET_DATA_DELETED (replaced by tablet a73a891fb6034f989d29a3f837303d84) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.455339 1042 tablet_service.cc:1505] Processing CreateTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b (DEFAULT_TABLE table=default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311 [id=14d3bbf59230416e97563c8371a9fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.462040 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2bb1075c4418eb10f4493f5e73c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.464186 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4cdf03acb92449d847aec897868b967" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.455722 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae561a995c94fadb288b6395c55a1bf (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.464707 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae561a995c94fadb288b6395c55a1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.455919 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.456089 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.456430 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet cf4934c3842a463e857262ceed37e0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 39dc5ebeb36043e0aa5955bacd06c2ef) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.456612 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet a61235da8c7a4f31b29d5d50c53c347d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.457002 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 13eeb2e8951b421e9d6f8f228a956e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.457163 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet a3a28a4aa361468c9fa9e72ccd6e7b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.459110 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2947083e3e447d5addae5effd186eb8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:07.469650 861 leader_election.cc:343] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.469745 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2947083e3e447d5addae5effd186eb8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.469822 861 leader_election.cc:304] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:07.470369 1353 raft_consensus.cc:2749] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:07.470535 718 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:07.471082 720 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.471212 720 leader_election.cc:304] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.471698 1461 raft_consensus.cc:2749] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.476222 1175 ts_tablet_manager.cc:1929] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.476392 1175 log.cc:1199] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/6426b5a4f5784b3a97475cb23199f916
I20251029 08:52:07.476754 1175 ts_tablet_manager.cc:1950] T 6426b5a4f5784b3a97475cb23199f916 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.477689 1175 tablet_replica.cc:333] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.477902 1175 raft_consensus.cc:2243] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:07.478382 1175 raft_consensus.cc:2272] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.479338 1175 ts_tablet_manager.cc:1916] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.479333 1438 raft_consensus.cc:493] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.479488 1438 raft_consensus.cc:515] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.479732 1176 ts_tablet_manager.cc:1929] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.479836 1176 log.cc:1199] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e4cdf03acb92449d847aec897868b967
I20251029 08:52:07.480199 1176 ts_tablet_manager.cc:1950] T e4cdf03acb92449d847aec897868b967 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.480264 1438 leader_election.cc:290] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:07.481663 996 leader_election.cc:343] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: cf4934c3842a463e857262ceed37e0bc
W20251029 08:52:07.482060 998 leader_election.cc:343] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: cf4934c3842a463e857262ceed37e0bc
I20251029 08:52:07.482173 998 leader_election.cc:304] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.482637 1438 raft_consensus.cc:2749] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.483204 1177 ts_tablet_manager.cc:1434] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20251029 08:52:07.485308 1177 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.486168 1183 tablet_bootstrap.cc:654] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.487875 1177 tablet_bootstrap.cc:654] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.488593 1176 tablet_replica.cc:333] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.488857 1176 raft_consensus.cc:2243] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.489050 1176 raft_consensus.cc:2272] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.489874 1176 ts_tablet_manager.cc:1916] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.487977 1175 ts_tablet_manager.cc:1929] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.490092 1175 log.cc:1199] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:07.490532 1175 ts_tablet_manager.cc:1950] T 13eeb2e8951b421e9d6f8f228a956e3d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.491410 1175 tablet_replica.cc:333] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.491725 1175 raft_consensus.cc:2243] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.492321 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]). Expected version 5 got 0
I20251029 08:52:07.492419 1175 raft_consensus.cc:2272] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.492900 1175 ts_tablet_manager.cc:1916] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.455389 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a20b64ff177241309e5d59e84f0bafa2 (table default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]) successfully deleted
I20251029 08:52:07.466010 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5a7a673ba4fd4abcbfc3493c38dcc257
I20251029 08:52:07.494293 594 catalog_manager.cc:6719] ReplaceTablet: tablet 5a7a673ba4fd4abcbfc3493c38dcc257 deleted and replaced by tablet 7b5051ff776e4975b917db0393a1f945
I20251029 08:52:07.494675 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1e141ef36f5541e5bba246df7272d2ec" } modify_external_catalogs: true
I20251029 08:52:07.494887 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1e141ef36f5541e5bba246df7272d2ec" } modify_external_catalogs: true
I20251029 08:52:07.468755 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1dd83e945754532a1f9694d8c9d2e0c" } modify_external_catalogs: true
I20251029 08:52:07.495924 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1dd83e945754532a1f9694d8c9d2e0c" } modify_external_catalogs: true
I20251029 08:52:07.470081 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_212e08a1f0ed463f9333cad96a91e4bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.497632 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_212e08a1f0ed463f9333cad96a91e4bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.481611 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 6426b5a4f5784b3a97475cb23199f916 (table default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]) successfully deleted
I20251029 08:52:07.498230 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 13eeb2e8951b421e9d6f8f228a956e3d (table default.Table_2e15dbeda3ab4c4b8473ccd12597b270 [id=407839d19657473883ad497873b78fb2]) successfully deleted
W20251029 08:52:07.498921 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ef0a7b31ced24388bc9d9964a42fa203 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0a7b31ced24388bc9d9964a42fa203 already in progress: creating tablet
I20251029 08:52:07.499193 1175 ts_tablet_manager.cc:1929] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:07.499213 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0ff4e625abf14fbd94a8102dfa5fc993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ff4e625abf14fbd94a8102dfa5fc993 already in progress: creating tablet
I20251029 08:52:07.499351 1175 log.cc:1199] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4e808bff7558472099587a0f95c002ff
W20251029 08:52:07.499614 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 23b534aaa51b4fad9dc3fc915de947f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b534aaa51b4fad9dc3fc915de947f1 already in progress: creating tablet
I20251029 08:52:07.499676 1175 ts_tablet_manager.cc:1950] T 4e808bff7558472099587a0f95c002ff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.500305 889 tablet_service.cc:1505] Processing CreateTablet for tablet 351047c725f8488abbd074bd40790b48 (DEFAULT_TABLE table=default.Table_af1eb2fb69584b62b01d7ec5d5711afd [id=de08f5f1950e4352a63c7d056bf4ce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.500708 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351047c725f8488abbd074bd40790b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.500862 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e4cdf03acb92449d847aec897868b967 (table default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]) successfully deleted
I20251029 08:52:07.501374 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d64b1a762f124195b4369c096587881d (table default.Table_8325dc8d37f147e79a2672a42a930de9 [id=12d36151f38e4a5783f82884f1eea461]) successfully deleted
I20251029 08:52:07.501736 902 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae561a995c94fadb288b6395c55a1bf (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.502137 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae561a995c94fadb288b6395c55a1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.502925 754 tablet_service.cc:1505] Processing CreateTablet for tablet b7da7e08b74f4a9aacedada42e839a11 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.503361 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7da7e08b74f4a9aacedada42e839a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.504207 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.504506 1183 tablet_bootstrap.cc:492] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.504694 1183 ts_tablet_manager.cc:1403] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20251029 08:52:07.505657 1183 raft_consensus.cc:359] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.505872 1183 raft_consensus.cc:385] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.505954 1183 raft_consensus.cc:740] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.506235 1183 consensus_queue.cc:260] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.506973 1183 ts_tablet_manager.cc:1434] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:07.507336 1183 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.508345 891 tablet_service.cc:1505] Processing CreateTablet for tablet b7da7e08b74f4a9aacedada42e839a11 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.509066 1183 tablet_bootstrap.cc:654] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.509465 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7da7e08b74f4a9aacedada42e839a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.509855 1179 ts_tablet_manager.cc:1929] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.510021 1179 log.cc:1199] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/6426b5a4f5784b3a97475cb23199f916
I20251029 08:52:07.510442 1179 ts_tablet_manager.cc:1950] T 6426b5a4f5784b3a97475cb23199f916 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.510838 897 tablet_service.cc:1505] Processing CreateTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.511292 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ba512bd4f04bd9aceb4cb210fe9689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.512676 763 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae561a995c94fadb288b6395c55a1bf (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.513126 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae561a995c94fadb288b6395c55a1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.513505 1176 ts_tablet_manager.cc:1929] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.513749 1176 log.cc:1199] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a2947083e3e447d5addae5effd186eb8
I20251029 08:52:07.514179 1176 ts_tablet_manager.cc:1950] T a2947083e3e447d5addae5effd186eb8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:07.514794 1177 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.514959 1177 ts_tablet_manager.cc:1403] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20251029 08:52:07.515853 1177 raft_consensus.cc:359] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.516093 1177 raft_consensus.cc:385] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.516188 1177 raft_consensus.cc:740] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.516484 1177 consensus_queue.cc:260] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.517144 1179 tablet_replica.cc:333] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.517372 1179 raft_consensus.cc:2243] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.517829 1179 raft_consensus.cc:2272] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.518462 1179 ts_tablet_manager.cc:1916] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.519375 1177 ts_tablet_manager.cc:1434] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:07.520423 1177 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.520867 1038 tablet_service.cc:1505] Processing CreateTablet for tablet b7da7e08b74f4a9aacedada42e839a11 (DEFAULT_TABLE table=default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.521225 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7da7e08b74f4a9aacedada42e839a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.522996 1177 tablet_bootstrap.cc:654] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.524550 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4e808bff7558472099587a0f95c002ff (table default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]) successfully deleted
I20251029 08:52:07.525017 1175 tablet_replica.cc:333] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.525313 1175 raft_consensus.cc:2243] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.525524 1175 raft_consensus.cc:2272] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.526005 1175 ts_tablet_manager.cc:1916] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.532724 1179 ts_tablet_manager.cc:1929] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.534330 748 tablet_service.cc:1505] Processing CreateTablet for tablet 351047c725f8488abbd074bd40790b48 (DEFAULT_TABLE table=default.Table_af1eb2fb69584b62b01d7ec5d5711afd [id=de08f5f1950e4352a63c7d056bf4ce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.533687 1042 tablet_service.cc:1505] Processing CreateTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.534642 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 6426b5a4f5784b3a97475cb23199f916 (table default.Table_70788044eecf4c7da477e61f7610e1c2 [id=963807fdd2de475f84f6e5b8a4e15e18]) successfully deleted
I20251029 08:52:07.534791 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351047c725f8488abbd074bd40790b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.534854 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ba512bd4f04bd9aceb4cb210fe9689. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.534874 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a1d5e1f68e4f0e8b78641bbdc98123 already in progress: creating tablet
W20251029 08:52:07.535033 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a8c5008d9df9419bb4b6d678c6dc27a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c5008d9df9419bb4b6d678c6dc27a6 already in progress: creating tablet
W20251029 08:52:07.535177 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e626994e96d54c0d913fdedc85fd1155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e626994e96d54c0d913fdedc85fd1155 already in progress: creating tablet
W20251029 08:52:07.535306 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc already in progress: creating tablet
W20251029 08:52:07.535450 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a20b64ff177241309e5d59e84f0bafa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20b64ff177241309e5d59e84f0bafa2 already in progress: creating tablet
W20251029 08:52:07.535594 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ef0a7b31ced24388bc9d9964a42fa203 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0a7b31ced24388bc9d9964a42fa203 already in progress: creating tablet
W20251029 08:52:07.535738 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 23b534aaa51b4fad9dc3fc915de947f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b534aaa51b4fad9dc3fc915de947f1 already in progress: creating tablet
I20251029 08:52:07.536067 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a2947083e3e447d5addae5effd186eb8 (table default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]) successfully deleted
W20251029 08:52:07.536240 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 363fcf8a3e0f4770a9743fddffa6495f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 363fcf8a3e0f4770a9743fddffa6495f already in progress: creating tablet
I20251029 08:52:07.536662 753 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.537134 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1c023b679b4c3298bf953b53d4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.539252 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.534087 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 351047c725f8488abbd074bd40790b48 (DEFAULT_TABLE table=default.Table_af1eb2fb69584b62b01d7ec5d5711afd [id=de08f5f1950e4352a63c7d056bf4ce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.540354 754 tablet_service.cc:1505] Processing CreateTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 (DEFAULT_TABLE table=default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.540514 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351047c725f8488abbd074bd40790b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.540740 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ba512bd4f04bd9aceb4cb210fe9689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.534278 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.547164 895 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.547637 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1c023b679b4c3298bf953b53d4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.549198 1179 log.cc:1199] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/06c08679f1e746538bfa2c80d4fbdba3
I20251029 08:52:07.549458 1353 consensus_queue.cc:1048] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:07.550702 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fd25eeb89e574297a92d8e978a79d768
I20251029 08:52:07.552615 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.553074 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1c023b679b4c3298bf953b53d4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:07.554042 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cc95257c937454aa6cd55b5a9394a56 consensus state has no leader
I20251029 08:52:07.554363 1177 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.554462 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_549035da52504bfa84a95642767fbe31" } modify_external_catalogs: true
I20251029 08:52:07.554669 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_549035da52504bfa84a95642767fbe31" } modify_external_catalogs: true
I20251029 08:52:07.557125 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cd7b287bef4d4e1da804f48f31500707" } new_table_name: "default.Table_7e364318a48e4b8eb94a67fcd126a67b" modify_external_catalogs: true
W20251029 08:52:07.558876 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9ed0a270b794e469d2fdbd0ea1d96f8 consensus state has no leader
I20251029 08:52:07.559264 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cd11e136b8054bdfb9c12527b0fc5777" } new_table_name: "default.Table_42cce27285944566a9037e99c90de248" modify_external_catalogs: true
I20251029 08:52:07.560216 1179 ts_tablet_manager.cc:1950] T 06c08679f1e746538bfa2c80d4fbdba3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.565220 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.565443 1179 tablet_replica.cc:333] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.566046 1179 raft_consensus.cc:2243] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.566126 748 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.566130 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 06c08679f1e746538bfa2c80d4fbdba3 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:07.566488 1179 raft_consensus.cc:2272] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:07.566625 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 363fcf8a3e0f4770a9743fddffa6495f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 363fcf8a3e0f4770a9743fddffa6495f already in progress: creating tablet
W20251029 08:52:07.566875 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0ff4e625abf14fbd94a8102dfa5fc993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ff4e625abf14fbd94a8102dfa5fc993 already in progress: creating tablet
W20251029 08:52:07.567128 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a7a673ba4fd4abcbfc3493c38dcc257 already in progress: creating tablet
W20251029 08:52:07.567492 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fd25eeb89e574297a92d8e978a79d768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd25eeb89e574297a92d8e978a79d768 already in progress: creating tablet
I20251029 08:52:07.567561 1179 ts_tablet_manager.cc:1916] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.572669 1179 ts_tablet_manager.cc:1929] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.572785 1179 log.cc:1199] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/cf4934c3842a463e857262ceed37e0bc
I20251029 08:52:07.573127 1179 ts_tablet_manager.cc:1950] T cf4934c3842a463e857262ceed37e0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.574040 1179 tablet_replica.cc:333] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.574240 1179 raft_consensus.cc:2243] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.574404 1179 raft_consensus.cc:2272] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.579265 1179 ts_tablet_manager.cc:1916] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.580752 1178 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.580924 1178 ts_tablet_manager.cc:1403] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.197s user 0.012s sys 0.048s
I20251029 08:52:07.581737 1178 raft_consensus.cc:359] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.581962 1178 raft_consensus.cc:385] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.582032 1178 raft_consensus.cc:740] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.582295 1178 consensus_queue.cc:260] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.582796 1178 ts_tablet_manager.cc:1434] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:07.583096 1178 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.584350 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet cf4934c3842a463e857262ceed37e0bc (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
I20251029 08:52:07.590010 1179 ts_tablet_manager.cc:1929] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.590173 1179 log.cc:1199] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a61235da8c7a4f31b29d5d50c53c347d
I20251029 08:52:07.590595 1179 ts_tablet_manager.cc:1950] T a61235da8c7a4f31b29d5d50c53c347d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.591784 1179 tablet_replica.cc:333] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.591969 1179 raft_consensus.cc:2243] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.592424 1179 raft_consensus.cc:2272] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.593024 1179 ts_tablet_manager.cc:1916] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.593067 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:07.593923 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b60d8ff960f148d586e443888f754174"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.594517 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b60d8ff960f148d586e443888f754174 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.596678 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0cde7b2972814447a6da2ec23d3b645f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.597178 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cde7b2972814447a6da2ec23d3b645f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.597470 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a61235da8c7a4f31b29d5d50c53c347d (table default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]) successfully deleted
W20251029 08:52:07.597738 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 13eeb2e8951b421e9d6f8f228a956e3d because the tablet was not found. No further retry: Not found: Tablet not found: 13eeb2e8951b421e9d6f8f228a956e3d
I20251029 08:52:07.597841 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.598682 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:07.599423 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_78b0f14f774845d5bd92e0902a2c79fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.599946 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b0f14f774845d5bd92e0902a2c79fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.601684 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f33c0bbf104046ccb86ff313f08da22a
I20251029 08:52:07.602298 588 catalog_manager.cc:6719] ReplaceTablet: tablet f33c0bbf104046ccb86ff313f08da22a deleted and replaced by tablet aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:07.602831 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8c5b5a48612f4ed4a391c6cd25320eea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.603356 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c5b5a48612f4ed4a391c6cd25320eea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.604971 597 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 5f2dd9d6d0854e03a8298625938065c5 (table default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c]). Expected version 1 got 0
I20251029 08:52:07.606736 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:07.606992 1183 tablet_bootstrap.cc:492] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.607137 1183 ts_tablet_manager.cc:1403] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.013s
I20251029 08:52:07.606973 589 catalog_manager.cc:5649] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:07.607481 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 1 got 0
I20251029 08:52:07.607714 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_7b45e52828bf4727815bfbb7c1ddb9c3 [id=63588e39919a481c856255fd848bbf98]). Expected version 1 got 0
I20251029 08:52:07.607870 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]). Expected version 5 got 0
I20251029 08:52:07.608055 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_cd11e136b8054bdfb9c12527b0fc5777 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 1 got 0
I20251029 08:52:07.608165 1183 raft_consensus.cc:359] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.608361 1179 ts_tablet_manager.cc:1929] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.608428 1183 raft_consensus.cc:385] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.608500 1179 log.cc:1199] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a3a28a4aa361468c9fa9e72ccd6e7b7f
I20251029 08:52:07.608565 1183 raft_consensus.cc:740] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.608916 1179 ts_tablet_manager.cc:1950] T a3a28a4aa361468c9fa9e72ccd6e7b7f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.608886 1183 consensus_queue.cc:260] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.609449 1183 ts_tablet_manager.cc:1434] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:07.586176 1178 tablet_bootstrap.cc:654] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.598747 763 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.611539 1183 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.612298 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.612709 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.613688 1183 tablet_bootstrap.cc:654] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.613745 1177 ts_tablet_manager.cc:1403] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20251029 08:52:07.613796 748 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:07.613853 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c572c68b32d443b2a019e1f4bf2d82fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c572c68b32d443b2a019e1f4bf2d82fe already in progress: creating tablet
I20251029 08:52:07.614120 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a3a28a4aa361468c9fa9e72ccd6e7b7f (table default.Table_08a5dc2ab2a94cbf8e92ad222fa6d165 [id=df4bb4cd3b3f46579ac59a716525c10b]) successfully deleted
W20251029 08:52:07.614423 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a7a673ba4fd4abcbfc3493c38dcc257 already in progress: creating tablet
I20251029 08:52:07.614495 902 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.614696 763 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:07.614817 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fd25eeb89e574297a92d8e978a79d768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd25eeb89e574297a92d8e978a79d768 already in progress: creating tablet
I20251029 08:52:07.615175 897 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:07.615255 861 consensus_peers.cc:597] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64b1a762f124195b4369c096587881d. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:07.617782 1175 ts_tablet_manager.cc:1929] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.617952 1175 log.cc:1199] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/904523f7164849139cddb1100e95b824
I20251029 08:52:07.618430 1175 ts_tablet_manager.cc:1950] T 904523f7164849139cddb1100e95b824 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.619155 902 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.620095 1178 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.620276 1178 ts_tablet_manager.cc:1403] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20251029 08:52:07.621086 1178 raft_consensus.cc:359] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.621296 1178 raft_consensus.cc:385] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.621399 1178 raft_consensus.cc:740] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.621649 1178 consensus_queue.cc:260] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.621740 1177 raft_consensus.cc:359] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.621891 1183 tablet_bootstrap.cc:492] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.621925 1177 raft_consensus.cc:385] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.622051 1183 ts_tablet_manager.cc:1403] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20251029 08:52:07.622223 1178 ts_tablet_manager.cc:1434] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251029 08:52:07.622337 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f33c0bbf104046ccb86ff313f08da22a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f33c0bbf104046ccb86ff313f08da22a already in progress: creating tablet
I20251029 08:52:07.622567 1178 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:07.622639 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bc9c221eaa9a4721ba25761206a13645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc9c221eaa9a4721ba25761206a13645 already in progress: creating tablet
W20251029 08:52:07.623021 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c572c68b32d443b2a019e1f4bf2d82fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c572c68b32d443b2a019e1f4bf2d82fe already in progress: creating tablet
I20251029 08:52:07.623059 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.623049 1183 raft_consensus.cc:359] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.623241 1183 raft_consensus.cc:385] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:07.623252 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bc9c221eaa9a4721ba25761206a13645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc9c221eaa9a4721ba25761206a13645 already in progress: creating tablet
I20251029 08:52:07.623327 1183 raft_consensus.cc:740] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:07.623487 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f33c0bbf104046ccb86ff313f08da22a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f33c0bbf104046ccb86ff313f08da22a already in progress: creating tablet
I20251029 08:52:07.623600 1183 consensus_queue.cc:260] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.624131 1183 ts_tablet_manager.cc:1434] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 08:52:07.624418 1178 tablet_bootstrap.cc:654] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.624833 1183 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.625260 1177 raft_consensus.cc:740] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.625609 1177 consensus_queue.cc:260] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.628441 1461 raft_consensus.cc:493] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.628603 1461 raft_consensus.cc:515] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.629878 1183 tablet_bootstrap.cc:654] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.634223 1461 leader_election.cc:290] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.635650 590 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f3677502ef124c01a5c33892d39bbca1
W20251029 08:52:07.636314 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc57cd5a8d3c42ba90f2011084e8b4bd consensus state has no leader
I20251029 08:52:07.636722 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.637329 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:07.639061 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8cb0c6c6a1c4877a89476d4822f48db consensus state has no leader
I20251029 08:52:07.640210 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_753225d529ba4e54b0b04b6388919367" } modify_external_catalogs: true
I20251029 08:52:07.640488 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_753225d529ba4e54b0b04b6388919367" } modify_external_catalogs: true
I20251029 08:52:07.642068 1178 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.642798 1178 ts_tablet_manager.cc:1403] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251029 08:52:07.644433 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71ba5e1248dc4d4a8a4d7bb05b661a3f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.645520 1178 raft_consensus.cc:359] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.645816 1178 raft_consensus.cc:385] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.645897 1178 raft_consensus.cc:740] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.646220 1178 consensus_queue.cc:260] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.648119 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_67b2c204ba81497680343eaa3b48b255" } new_table_name: "default.Table_c36b90fc430a4a89ba87469be66cb2a5" modify_external_catalogs: true
W20251029 08:52:07.650789 720 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.651860 1178 ts_tablet_manager.cc:1434] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:07.653623 1178 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.659991 1448 raft_consensus.cc:493] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.660279 1448 raft_consensus.cc:515] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.661118 1448 leader_election.cc:290] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.661839 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.662539 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:07.662843 782 raft_consensus.cc:2468] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:07.663632 861 leader_election.cc:343] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.664186 861 leader_election.cc:304] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.665076 1448 raft_consensus.cc:2804] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.665238 1448 raft_consensus.cc:493] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.665354 1448 raft_consensus.cc:3060] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.666092 1177 ts_tablet_manager.cc:1434] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251029 08:52:07.667448 1178 tablet_bootstrap.cc:654] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.672957 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71ba5e1248dc4d4a8a4d7bb05b661a3f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:07.673823 718 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.674046 718 leader_election.cc:304] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.674592 1461 raft_consensus.cc:2749] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.676178 1178 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.676339 1178 ts_tablet_manager.cc:1403] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20251029 08:52:07.677183 1178 raft_consensus.cc:359] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.677197 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 904523f7164849139cddb1100e95b824 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
I20251029 08:52:07.677376 1178 raft_consensus.cc:385] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.677461 1178 raft_consensus.cc:740] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.677681 1178 consensus_queue.cc:260] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.678442 1178 ts_tablet_manager.cc:1434] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:07.678778 1178 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.678931 1177 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:07.680806 1178 tablet_bootstrap.cc:654] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.680928 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 57e2b19a227740a28ae601d77e62597b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e2b19a227740a28ae601d77e62597b already in progress: creating tablet
I20251029 08:52:07.682137 1175 tablet_replica.cc:333] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:07.682878 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 63b08840d0fa40428c61a05b277ad750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63b08840d0fa40428c61a05b277ad750 already in progress: creating tablet
W20251029 08:52:07.683177 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a20f26d0f7874ccb960ffe4d27e854e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20f26d0f7874ccb960ffe4d27e854e1 already in progress: creating tablet
I20251029 08:52:07.683920 1438 raft_consensus.cc:493] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.684165 1438 raft_consensus.cc:515] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.684350 1175 raft_consensus.cc:2243] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:07.684748 1175 pending_rounds.cc:70] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:07.684966 1438 leader_election.cc:290] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.685134 1175 pending_rounds.cc:77] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037587030888448 op_type: NO_OP noop_request { }
I20251029 08:52:07.685281 1175 raft_consensus.cc:2889] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:07.685627 1175 raft_consensus.cc:2272] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:07.686093 998 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:07.686416 1175 ts_tablet_manager.cc:1916] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.688331 1177 tablet_bootstrap.cc:654] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.688473 996 leader_election.cc:343] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:07.688633 996 leader_election.cc:304] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.689226 1438 raft_consensus.cc:2749] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.692035 1178 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.692224 1178 ts_tablet_manager.cc:1403] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s
I20251029 08:52:07.693027 1178 raft_consensus.cc:359] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.693274 1178 raft_consensus.cc:385] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.693354 1178 raft_consensus.cc:740] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.693635 1178 consensus_queue.cc:260] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.694133 1178 ts_tablet_manager.cc:1434] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:07.694480 1178 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.696281 1178 tablet_bootstrap.cc:654] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.704319 1175 ts_tablet_manager.cc:1929] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.704480 1175 log.cc:1199] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d64b1a762f124195b4369c096587881d
I20251029 08:52:07.716806 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c3e50018f414f4b88e0e5d90cb7c925 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.718482 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_74a8940fc31c4282944999b26eb6ce83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.719142 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74a8940fc31c4282944999b26eb6ce83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.720995 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d5036ccd9a194fcf828317d2b8060b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.721643 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5036ccd9a194fcf828317d2b8060b73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.723366 597 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03] alter complete (version 2)
I20251029 08:52:07.730788 1178 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.730962 1178 ts_tablet_manager.cc:1403] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20251029 08:52:07.732823 1178 raft_consensus.cc:359] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.733022 1178 raft_consensus.cc:385] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.733120 1178 raft_consensus.cc:740] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.740960 1178 consensus_queue.cc:260] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.744428 1178 ts_tablet_manager.cc:1434] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20251029 08:52:07.744882 1178 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.746312 1461 raft_consensus.cc:493] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.746475 1461 raft_consensus.cc:515] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.746803 1178 tablet_bootstrap.cc:654] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.747655 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a2b37e45e444caa3f5117ae9a25304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.748139 1179 tablet_replica.cc:333] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.748442 1179 raft_consensus.cc:2243] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.748669 1179 raft_consensus.cc:2272] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.749073 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b11998f3ebf3402ab3f729d04fd5e0ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.749722 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 904523f7164849139cddb1100e95b824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.750644 1179 ts_tablet_manager.cc:1916] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.760725 1461 leader_election.cc:290] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:07.761238 720 leader_election.cc:343] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.761835 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.761930 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e08d8a7733e048299c27863d38207448" } new_table_name: "default.Table_61c7054042014cdb9fde5c728083e742" modify_external_catalogs: true
I20251029 08:52:07.762089 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.762279 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.762936 1176 tablet_replica.cc:333] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.763121 1176 raft_consensus.cc:2243] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.763301 1176 raft_consensus.cc:2272] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.763645 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9b80aae43bb845d1bb98687e0e6116f6" } modify_external_catalogs: true
I20251029 08:52:07.763890 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9b80aae43bb845d1bb98687e0e6116f6" } modify_external_catalogs: true
I20251029 08:52:07.764272 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f159f31991334c3cac52a8b4818345e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:07.764573 1176 ts_tablet_manager.cc:1916] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:07.765620 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f159f31991334c3cac52a8b4818345e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.767771 920 raft_consensus.cc:1275] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Refusing update from remote peer 7a8d80098fcb40ada903c3b2bacc2ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:07.768579 1366 consensus_queue.cc:1048] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:07.774076 720 consensus_peers.cc:597] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:07.778982 1562 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:07.781054 891 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.781404 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b11998f3ebf3402ab3f729d04fd5e0ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.781622 923 raft_consensus.cc:2468] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:07.781857 1448 raft_consensus.cc:515] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.783645 718 leader_election.cc:304] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.783869 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:07.784087 782 raft_consensus.cc:3060] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.784391 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:07.785133 861 leader_election.cc:343] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.785370 1448 leader_election.cc:290] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.785703 1461 raft_consensus.cc:2804] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.785899 1461 raft_consensus.cc:493] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.786031 1461 raft_consensus.cc:3060] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.787688 782 raft_consensus.cc:2468] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:07.789016 1563 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:07.789215 861 leader_election.cc:304] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.791026 1461 raft_consensus.cc:515] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.792596 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b11998f3ebf3402ab3f729d04fd5e0ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:07.792865 923 raft_consensus.cc:3060] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.796640 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b11998f3ebf3402ab3f729d04fd5e0ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:07.797425 720 leader_election.cc:343] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.797700 1461 leader_election.cc:290] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.798280 1176 ts_tablet_manager.cc:1929] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.802243 1448 raft_consensus.cc:2804] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.802525 1175 ts_tablet_manager.cc:1950] T d64b1a762f124195b4369c096587881d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:07.807214 1179 ts_tablet_manager.cc:1929] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.807560 1176 log.cc:1199] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/57e2b19a227740a28ae601d77e62597b
I20251029 08:52:07.807821 1179 log.cc:1199] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/16a2b37e45e444caa3f5117ae9a25304
I20251029 08:52:07.808506 1179 ts_tablet_manager.cc:1950] T 16a2b37e45e444caa3f5117ae9a25304 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.808923 1176 ts_tablet_manager.cc:1950] T 57e2b19a227740a28ae601d77e62597b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:07.810624 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251029 08:52:07.817147 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:07.818953 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_e08d8a7733e048299c27863d38207448 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 1 got 0
W20251029 08:52:07.819543 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 084a659932ec4996a6d6b160d67efdd0 consensus state has no leader
I20251029 08:52:07.819955 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bd00dc7aa1464015aafad884320e9740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.820593 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd00dc7aa1464015aafad884320e9740 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.821343 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.823756 1176 tablet_replica.cc:333] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:07.823901 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 57e2b19a227740a28ae601d77e62597b (table default.Table_ab2c1cce4615467595cb78eb6ca1b4ec [id=d283eaf060754468a34dfd1849f618ad]) successfully deleted
I20251029 08:52:07.824193 1448 raft_consensus.cc:697] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:07.825109 596 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:07.825301 1353 raft_consensus.cc:493] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.825462 1353 raft_consensus.cc:515] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.827445 1353 leader_election.cc:290] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.827641 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_90db61e4d27e4b409e948b3f33c765e9" } new_table_name: "default.Table_671a7306577e49c98f6ec7c4317f2f31" modify_external_catalogs: true
W20251029 08:52:07.835456 583 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b -> Peer f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435): Couldn't send request to peer f718bd0f00004f20a89d028a2fa8e6b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251029 08:52:07.838460 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:07.839119 591 catalog_manager.cc:6719] ReplaceTablet: tablet 7c3e50018f414f4b88e0e5d90cb7c925 deleted and replaced by tablet cd6208b92fe94dc881cf45b4627d68f2
I20251029 08:52:07.839547 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b9b1c4a272214856ab50337015cdbd7b" } modify_external_catalogs: true
I20251029 08:52:07.839787 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b9b1c4a272214856ab50337015cdbd7b" } modify_external_catalogs: true
I20251029 08:52:07.841004 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1e96e0acdb10410a89f8f34e1f44deff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.841621 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e96e0acdb10410a89f8f34e1f44deff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.809181 1438 raft_consensus.cc:493] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.841975 1438 raft_consensus.cc:515] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.842833 1438 leader_election.cc:290] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:07.821842 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.843557 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e" } modify_external_catalogs: true
I20251029 08:52:07.843734 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e" } modify_external_catalogs: true
I20251029 08:52:07.830087 1183 tablet_bootstrap.cc:492] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.843997 1183 ts_tablet_manager.cc:1403] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.220s user 0.030s sys 0.067s
I20251029 08:52:07.845283 1183 raft_consensus.cc:359] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.845721 1183 raft_consensus.cc:385] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.845827 1183 raft_consensus.cc:740] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.846118 1183 consensus_queue.cc:260] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.848731 1183 ts_tablet_manager.cc:1434] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 08:52:07.848905 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 9702a7d8b8304b49a250c129ecf28303 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.849223 1183 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.850510 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.852433 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_25994a51dae84fef8fcd0519119cf2af" } new_table_name: "default.Table_2073f0e27ae9417eb3f4273468277ff0" modify_external_catalogs: true
I20251029 08:52:07.852813 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0f93256544fb4b56adbbcaf8c50ecb42" } modify_external_catalogs: true
I20251029 08:52:07.852984 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0f93256544fb4b56adbbcaf8c50ecb42" } modify_external_catalogs: true
I20251029 08:52:07.854230 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6e61b44def8f4ea283b27494c59f9926" } new_table_name: "default.Table_58fa7040e0e2467588dd8b5a0a2eba4b" modify_external_catalogs: true
W20251029 08:52:07.856247 861 leader_election.cc:343] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: a61235da8c7a4f31b29d5d50c53c347d
W20251029 08:52:07.856729 861 leader_election.cc:343] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a61235da8c7a4f31b29d5d50c53c347d
I20251029 08:52:07.856868 861 leader_election.cc:304] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:07.857111 1353 raft_consensus.cc:2749] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.859699 1179 tablet_replica.cc:333] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.860611 1183 tablet_bootstrap.cc:654] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.861900 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.862032 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 51bb01c123c44fb69b2f174fca5d7a13 with delete_type TABLET_DATA_DELETED (replaced by tablet c4e1b6b6359b437aa295b473fa42dce1) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.862439 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 16a2b37e45e444caa3f5117ae9a25304 (table default.Table_823727b7e1194e56a35d64f1d58d761a [id=bfa9e40683954f66b64aaffd52ee446e]) successfully deleted
I20251029 08:52:07.859961 1179 raft_consensus.cc:2243] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:07.863632 1179 pending_rounds.cc:70] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:07.863719 1179 pending_rounds.cc:77] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037590316281856 op_type: NO_OP noop_request { }
I20251029 08:52:07.863864 1179 raft_consensus.cc:2889] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:07.864059 1179 raft_consensus.cc:2272] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.864940 1179 ts_tablet_manager.cc:1916] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.866370 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.867413 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d64b1a762f124195b4369c096587881d (table default.Table_8325dc8d37f147e79a2672a42a930de9 [id=12d36151f38e4a5783f82884f1eea461]) successfully deleted
W20251029 08:52:07.868669 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 351047c725f8488abbd074bd40790b48 (table default.Table_af1eb2fb69584b62b01d7ec5d5711afd [id=de08f5f1950e4352a63c7d056bf4ce86], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 351047c725f8488abbd074bd40790b48 consensus state has no leader
I20251029 08:52:07.869077 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:07.869648 923 raft_consensus.cc:2468] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
W20251029 08:52:07.869742 861 consensus_peers.cc:597] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:07.870265 1177 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.870756 1177 ts_tablet_manager.cc:1403] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.194s user 0.015s sys 0.030s
I20251029 08:52:07.867693 1448 consensus_queue.cc:237] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.874411 1177 raft_consensus.cc:359] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.875119 1068 raft_consensus.cc:3060] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.828478 1461 raft_consensus.cc:493] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.875458 1461 raft_consensus.cc:515] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.876212 1461 leader_election.cc:290] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.878957 1068 raft_consensus.cc:1275] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:07.834256 1366 raft_consensus.cc:493] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:07.879463 1366 raft_consensus.cc:515] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.880126 1366 leader_election.cc:290] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.846180 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f2dd9d6d0854e03a8298625938065c5" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:07.863147 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:07.881175 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7fdbe6681894ffa8264cdfa486f16e6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:07.881175 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cf906c73dc84403bcc3dece74a822bd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:07.881844 720 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:07.882098 720 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:07.882982 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e626994e96d54c0d913fdedc85fd1155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e626994e96d54c0d913fdedc85fd1155 already in progress: creating tablet
I20251029 08:52:07.870180 718 leader_election.cc:304] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.873136 783 raft_consensus.cc:1275] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:07.884140 998 leader_election.cc:304] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.884454 1438 raft_consensus.cc:2749] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.884603 1366 raft_consensus.cc:2804] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.884765 1366 raft_consensus.cc:697] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:07.885030 1366 consensus_queue.cc:237] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.887993 1178 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.888228 1178 ts_tablet_manager.cc:1403] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.143s user 0.007s sys 0.044s
I20251029 08:52:07.888563 1177 raft_consensus.cc:385] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.888693 1177 raft_consensus.cc:740] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.888885 1433 consensus_queue.cc:1048] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 08:52:07.889130 1178 raft_consensus.cc:359] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.889349 1178 raft_consensus.cc:385] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.889422 1178 raft_consensus.cc:740] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.889458 1177 consensus_queue.cc:260] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.889681 1178 consensus_queue.cc:260] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.889834 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7fdbe6681894ffa8264cdfa486f16e6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.890161 1178 ts_tablet_manager.cc:1434] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:07.890511 1178 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.892925 1178 tablet_bootstrap.cc:654] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.898185 920 raft_consensus.cc:2468] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:07.898810 718 leader_election.cc:304] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.899992 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cf906c73dc84403bcc3dece74a822bd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:07.900049 1433 consensus_queue.cc:1048] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20251029 08:52:07.903026 718 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.903203 718 leader_election.cc:304] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:07.903649 1461 raft_consensus.cc:2804] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:07.903769 1461 raft_consensus.cc:493] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:07.903854 1461 raft_consensus.cc:3060] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.907021 1461 raft_consensus.cc:515] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.908457 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7fdbe6681894ffa8264cdfa486f16e6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:07.909216 720 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:07.910275 1461 leader_election.cc:290] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:07.910501 1461 raft_consensus.cc:2749] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:07.912757 1183 tablet_bootstrap.cc:492] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.912765 1178 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.912926 1178 ts_tablet_manager.cc:1403] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251029 08:52:07.913017 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7fdbe6681894ffa8264cdfa486f16e6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:07.913285 923 raft_consensus.cc:3060] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.913858 1178 raft_consensus.cc:359] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.914085 1178 raft_consensus.cc:385] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.914166 1178 raft_consensus.cc:740] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.914418 1178 consensus_queue.cc:260] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:07.914440 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 276) took 1014 ms. Trace:
I20251029 08:52:07.913650 1177 ts_tablet_manager.cc:1434] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251029 08:52:07.914950 1175 rpcz_store.cc:276] 1029 08:52:06.885106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:06.885141 (+ 35us) service_pool.cc:225] Handling call
1029 08:52:07.874755 (+989614us) ts_tablet_manager.cc:1151] Deleting tablet c9a1d5e1f68e4f0e8b78641bbdc98123
1029 08:52:07.874766 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:07.899335 (+ 24569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":989441}
I20251029 08:52:07.916397 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 277) took 1030 ms. Trace:
I20251029 08:52:07.916538 1175 rpcz_store.cc:276] 1029 08:52:06.885427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:06.885465 (+ 38us) service_pool.cc:225] Handling call
1029 08:52:07.916199 (+1030734us) ts_tablet_manager.cc:1151] Deleting tablet a8c5008d9df9419bb4b6d678c6dc27a6
1029 08:52:07.916210 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:07.916351 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030591}
I20251029 08:52:07.916924 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 271) took 1034 ms. Trace:
I20251029 08:52:07.917025 1175 rpcz_store.cc:276] 1029 08:52:06.882953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:06.943777 (+ 60824us) service_pool.cc:225] Handling call
1029 08:52:07.916776 (+972999us) ts_tablet_manager.cc:1151] Deleting tablet abc307b14aff4c5e937a5151aab1dfb9
1029 08:52:07.916785 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:07.916886 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":937368}
I20251029 08:52:07.917418 1175 tablet_replica.cc:333] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:07.917665 1175 raft_consensus.cc:2243] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.917836 1175 raft_consensus.cc:2272] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.917845 1178 ts_tablet_manager.cc:1434] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251029 08:52:07.918110 923 raft_consensus.cc:2468] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:07.918182 1178 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.918411 1175 ts_tablet_manager.cc:1916] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.920881 1177 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:07.921017 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a1d5e1f68e4f0e8b78641bbdc98123 already in progress: creating tablet
W20251029 08:52:07.921300 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a8c5008d9df9419bb4b6d678c6dc27a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c5008d9df9419bb4b6d678c6dc27a6 already in progress: creating tablet
W20251029 08:52:07.921514 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet abc307b14aff4c5e937a5151aab1dfb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc307b14aff4c5e937a5151aab1dfb9 already in progress: creating tablet
W20251029 08:52:07.921730 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc already in progress: creating tablet
I20251029 08:52:07.921988 1183 ts_tablet_manager.cc:1403] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.004s
I20251029 08:52:07.922102 1179 ts_tablet_manager.cc:1929] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:07.922459 1179 log.cc:1199] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/904523f7164849139cddb1100e95b824
I20251029 08:52:07.922773 1177 tablet_bootstrap.cc:654] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.923135 1179 ts_tablet_manager.cc:1950] T 904523f7164849139cddb1100e95b824 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.923192 1178 tablet_bootstrap.cc:654] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.924868 1179 tablet_replica.cc:333] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.925340 1179 raft_consensus.cc:2243] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.925626 1179 raft_consensus.cc:2272] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.926476 1179 ts_tablet_manager.cc:1916] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.926625 1178 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.926801 1178 ts_tablet_manager.cc:1403] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20251029 08:52:07.927623 1178 raft_consensus.cc:359] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.927845 1178 raft_consensus.cc:385] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.928613 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 904523f7164849139cddb1100e95b824 (table default.Table_076b5e5e98044d93b9c0d75227b7e0c3 [id=8da84d5e1133484d8e075abbb6a5d2da]) successfully deleted
W20251029 08:52:07.928921 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d30183814f5e4f2ab9d71fe493eb801c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30183814f5e4f2ab9d71fe493eb801c already in progress: creating tablet
W20251029 08:52:07.929116 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7c3e50018f414f4b88e0e5d90cb7c925 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c3e50018f414f4b88e0e5d90cb7c925 already in progress: creating tablet
I20251029 08:52:07.923018 1183 raft_consensus.cc:359] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.929852 1183 raft_consensus.cc:385] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.929976 1183 raft_consensus.cc:740] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.930385 1183 consensus_queue.cc:260] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.931064 1178 raft_consensus.cc:740] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.931412 1178 consensus_queue.cc:260] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.931833 1183 ts_tablet_manager.cc:1434] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:07.931917 1177 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:07.932092 1177 ts_tablet_manager.cc:1403] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20251029 08:52:07.932734 1178 ts_tablet_manager.cc:1434] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251029 08:52:07.933072 1178 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.934368 1183 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.934693 718 leader_election.cc:304] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:07.934978 1461 raft_consensus.cc:2804] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:07.937580 1068 raft_consensus.cc:3060] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:07.937692 1178 tablet_bootstrap.cc:654] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.934304 1177 raft_consensus.cc:359] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.943905 1177 raft_consensus.cc:385] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.945458 1178 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:07.945662 1178 ts_tablet_manager.cc:1403] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251029 08:52:07.946724 1178 raft_consensus.cc:359] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.946950 1178 raft_consensus.cc:385] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.947060 1178 raft_consensus.cc:740] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.947001 1177 raft_consensus.cc:740] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.947712 1183 tablet_bootstrap.cc:654] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:07.943399 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b5051ff776e4975b917db0393a1f945 has no consensus state
I20251029 08:52:07.947476 1177 consensus_queue.cc:260] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.949700 1178 consensus_queue.cc:260] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.950903 1178 ts_tablet_manager.cc:1434] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 08:52:07.951243 1178 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:07.951673 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet bb6b86f39c994ea5930a6a970830093f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:07.957211 1461 raft_consensus.cc:697] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
W20251029 08:52:07.958289 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5385fef404a4463595074f0c8ff822d7 consensus state has no leader
I20251029 08:52:07.958634 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 61ff3274c7104c73bd35768c229833b1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:07.958702 1461 consensus_queue.cc:237] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:07.960321 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 09759a50a82f4b758449da9fbbc1f72d (table default.Table_0f93256544fb4b56adbbcaf8c50ecb42 [id=dc9c30acd58449e4bcb9217b628b59dc]). Expected version 1 got 0
I20251029 08:52:07.960544 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (table default.Table_b9b1c4a272214856ab50337015cdbd7b [id=4856bf7f477c47d58060d1728c544aa0]). Expected version 1 got 0
I20251029 08:52:07.962260 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 933817bf2058464f837ba96921c4ed63 from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:07.964495 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 898a4f58306b453e9abca2a73082bb96 consensus state has no leader
I20251029 08:52:07.964942 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6ba990fb812943a5bd7c7588dbb684dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:07.965528 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ba990fb812943a5bd7c7588dbb684dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:07.967046 1179 ts_tablet_manager.cc:1929] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:07.967185 1179 log.cc:1199] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/51bb01c123c44fb69b2f174fca5d7a13
I20251029 08:52:07.967568 1178 tablet_bootstrap.cc:654] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.967573 1179 ts_tablet_manager.cc:1950] T 51bb01c123c44fb69b2f174fca5d7a13 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:07.968480 1038 tablet_service.cc:1505] Processing CreateTablet for tablet de78ea608c904aeea7f7145b9c80f45b (DEFAULT_TABLE table=default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.968945 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de78ea608c904aeea7f7145b9c80f45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.970489 1183 tablet_bootstrap.cc:492] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:07.970659 1183 ts_tablet_manager.cc:1403] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20251029 08:52:07.972455 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 (DEFAULT_TABLE table=default.Table_32a7c0c8b359467995851e70b6448f62 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.972915 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ab9df6f768410eb4f66cffdcd68df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.974365 596 catalog_manager.cc:5649] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:07.974869 596 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 2 got 0
I20251029 08:52:07.976701 1040 tablet_service.cc:1505] Processing CreateTablet for tablet c66cbb495f5142d79a674f7d5adc505e (DEFAULT_TABLE table=default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.977234 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cbb495f5142d79a674f7d5adc505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.978859 1183 raft_consensus.cc:359] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.979012 1179 tablet_replica.cc:333] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:07.979086 1183 raft_consensus.cc:385] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:07.979208 1183 raft_consensus.cc:740] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:07.979271 1179 raft_consensus.cc:2243] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:07.979442 1179 raft_consensus.cc:2272] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:07.979568 1183 consensus_queue.cc:260] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:07.980350 1183 ts_tablet_manager.cc:1434] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20251029 08:52:07.980549 1179 ts_tablet_manager.cc:1916] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:07.980721 1183 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:07.982753 1183 tablet_bootstrap.cc:654] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:07.986749 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (DEFAULT_TABLE table=default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.987205 1043 tablet_service.cc:1505] Processing CreateTablet for tablet fc04616301354cfe873c825a9f5d7ab6 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.987672 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (DEFAULT_TABLE table=default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.988476 1049 tablet_service.cc:1505] Processing CreateTablet for tablet aafa846cb2144cec9ce4b28e61d91961 (DEFAULT_TABLE table=default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.988703 1036 tablet_service.cc:1505] Processing CreateTablet for tablet fe384ca428974ac5ba78e6506a1a8832 (DEFAULT_TABLE table=default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.988901 1051 tablet_service.cc:1505] Processing CreateTablet for tablet eed66c3fa3f848df905a2a31fd553474 (DEFAULT_TABLE table=default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.986804 1044 tablet_service.cc:1505] Processing CreateTablet for tablet bbc4345cabbc476485b6a040a076ddaf (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.989616 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 69d27944f3c14a6eb8331a904d6faa40 (DEFAULT_TABLE table=default.Table_2ec426aafdb6470a85870622c71f3f64 [id=ad2b093021204f7eaf35330fd0675654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.990339 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 51bb01c123c44fb69b2f174fca5d7a13 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
I20251029 08:52:07.993906 763 tablet_service.cc:1505] Processing CreateTablet for tablet c66cbb495f5142d79a674f7d5adc505e (DEFAULT_TABLE table=default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.994413 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cbb495f5142d79a674f7d5adc505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.994531 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.994817 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafa846cb2144cec9ce4b28e61d91961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.995025 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe384ca428974ac5ba78e6506a1a8832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.995260 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed66c3fa3f848df905a2a31fd553474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:07.996693 748 tablet_service.cc:1505] Processing CreateTablet for tablet de78ea608c904aeea7f7145b9c80f45b (DEFAULT_TABLE table=default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:07.997143 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de78ea608c904aeea7f7145b9c80f45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.000180 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.000418 749 tablet_service.cc:1505] Processing CreateTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (DEFAULT_TABLE table=default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.000689 753 tablet_service.cc:1505] Processing CreateTablet for tablet bbc4345cabbc476485b6a040a076ddaf (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.000893 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.001226 757 tablet_service.cc:1505] Processing CreateTablet for tablet fc04616301354cfe873c825a9f5d7ab6 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.001603 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc04616301354cfe873c825a9f5d7ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.000422 754 tablet_service.cc:1505] Processing CreateTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 (DEFAULT_TABLE table=default.Table_32a7c0c8b359467995851e70b6448f62 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.003046 759 tablet_service.cc:1505] Processing CreateTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (DEFAULT_TABLE table=default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.004052 746 tablet_service.cc:1505] Processing CreateTablet for tablet eed66c3fa3f848df905a2a31fd553474 (DEFAULT_TABLE table=default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.004189 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d27944f3c14a6eb8331a904d6faa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.004638 756 tablet_service.cc:1505] Processing CreateTablet for tablet fe384ca428974ac5ba78e6506a1a8832 (DEFAULT_TABLE table=default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.004925 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc4345cabbc476485b6a040a076ddaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.005576 750 tablet_service.cc:1505] Processing CreateTablet for tablet aafa846cb2144cec9ce4b28e61d91961 (DEFAULT_TABLE table=default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.008172 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc04616301354cfe873c825a9f5d7ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.010893 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafa846cb2144cec9ce4b28e61d91961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.012195 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc4345cabbc476485b6a040a076ddaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.013449 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ab9df6f768410eb4f66cffdcd68df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.035332 1433 consensus_queue.cc:1048] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.078s
I20251029 08:52:08.036267 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.040653 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed66c3fa3f848df905a2a31fd553474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.041378 891 tablet_service.cc:1505] Processing CreateTablet for tablet c66cbb495f5142d79a674f7d5adc505e (DEFAULT_TABLE table=default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.041905 1448 raft_consensus.cc:493] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.042150 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cbb495f5142d79a674f7d5adc505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.045729 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe384ca428974ac5ba78e6506a1a8832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.047951 1179 ts_tablet_manager.cc:1929] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.048116 1179 log.cc:1199] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bb6b86f39c994ea5930a6a970830093f
I20251029 08:52:08.048480 1179 ts_tablet_manager.cc:1950] T bb6b86f39c994ea5930a6a970830093f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:08.049554 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bb6b86f39c994ea5930a6a970830093f (table default.Table_73193142c4d04b1f83e12f1ac807a71f [id=2f82075a67804ac48d6817dd2924b9a8]) successfully deleted
I20251029 08:52:08.058640 893 tablet_service.cc:1505] Processing CreateTablet for tablet fe384ca428974ac5ba78e6506a1a8832 (DEFAULT_TABLE table=default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.059322 894 tablet_service.cc:1505] Processing CreateTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (DEFAULT_TABLE table=default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.062837 1183 tablet_bootstrap.cc:492] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.063042 1183 ts_tablet_manager.cc:1403] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.002s
I20251029 08:52:08.063977 1183 raft_consensus.cc:359] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.064224 1183 raft_consensus.cc:385] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.064327 1183 raft_consensus.cc:740] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.064594 1183 consensus_queue.cc:260] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.065157 1183 ts_tablet_manager.cc:1434] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:08.065557 1183 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.065708 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:08.066756 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b1c17c659dfe49f8835fb51a638b7b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.067391 1183 tablet_bootstrap.cc:654] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:08.067466 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1c17c659dfe49f8835fb51a638b7b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.068269 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:08.068795 593 catalog_manager.cc:6719] ReplaceTablet: tablet 9702a7d8b8304b49a250c129ecf28303 deleted and replaced by tablet 7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:08.069156 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5f9765183d954404b485e1a99c508874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.069342 1176 raft_consensus.cc:2243] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:08.069712 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f9765183d954404b485e1a99c508874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.069844 1176 pending_rounds.cc:70] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:08.073287 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7792ee223dbe43aa82990541caced045
I20251029 08:52:08.074098 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dd22da054ccc4c1f87f65c41649bd1a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.074693 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd22da054ccc4c1f87f65c41649bd1a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.075750 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61ff3274c7104c73bd35768c229833b1
I20251029 08:52:08.076294 590 catalog_manager.cc:6719] ReplaceTablet: tablet 61ff3274c7104c73bd35768c229833b1 deleted and replaced by tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:08.076655 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f6ce198ce16b46249069916455fe0165" } modify_external_catalogs: true
I20251029 08:52:08.076849 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f6ce198ce16b46249069916455fe0165" } modify_external_catalogs: true
I20251029 08:52:08.084896 1183 tablet_bootstrap.cc:492] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.084849 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 09759a50a82f4b758449da9fbbc1f72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.085062 1183 ts_tablet_manager.cc:1403] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20251029 08:52:08.085549 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 9702a7d8b8304b49a250c129ecf28303 with delete_type TABLET_DATA_DELETED (replaced by tablet 7be4bcf9e3324bd884e946fae20648fd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.085898 1183 raft_consensus.cc:359] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.086105 1183 raft_consensus.cc:385] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.086151 1176 pending_rounds.cc:77] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037592922128384 op_type: NO_OP noop_request { }
I20251029 08:52:08.086216 1183 raft_consensus.cc:740] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.086319 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.086494 1183 consensus_queue.cc:260] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.086946 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ff3274c7104c73bd35768c229833b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c1a8a1e4cc14f1aa6e4f91174f45127) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.087018 1183 ts_tablet_manager.cc:1434] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:08.089645 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 933817bf2058464f837ba96921c4ed63
I20251029 08:52:08.090276 594 catalog_manager.cc:6719] ReplaceTablet: tablet 933817bf2058464f837ba96921c4ed63 deleted and replaced by tablet e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:08.090960 1176 raft_consensus.cc:2889] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:08.091149 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e1f5ac232fb544b3a9e3397ef1472c07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.091372 1176 raft_consensus.cc:2272] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:08.091910 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1f5ac232fb544b3a9e3397ef1472c07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.088923 1179 tablet_replica.cc:333] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:08.093166 1179 raft_consensus.cc:2243] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:08.093174 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 09759a50a82f4b758449da9fbbc1f72d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09759a50a82f4b758449da9fbbc1f72d already in progress: creating tablet
I20251029 08:52:08.093433 1179 raft_consensus.cc:2272] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:08.093510 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9702a7d8b8304b49a250c129ecf28303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9702a7d8b8304b49a250c129ecf28303 already in progress: creating tablet
W20251029 08:52:08.093719 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7792ee223dbe43aa82990541caced045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7792ee223dbe43aa82990541caced045 already in progress: creating tablet
I20251029 08:52:08.093986 1179 ts_tablet_manager.cc:1916] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.094316 888 tablet_service.cc:1505] Processing CreateTablet for tablet fc04616301354cfe873c825a9f5d7ab6 (DEFAULT_TABLE table=default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.094658 896 tablet_service.cc:1505] Processing CreateTablet for tablet eed66c3fa3f848df905a2a31fd553474 (DEFAULT_TABLE table=default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.094828 889 tablet_service.cc:1505] Processing CreateTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (DEFAULT_TABLE table=default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.094985 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe384ca428974ac5ba78e6506a1a8832. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:08.087579 1582 consensus_queue.cc:1175] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251029 08:52:08.092911 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 933817bf2058464f837ba96921c4ed63 with delete_type TABLET_DATA_DELETED (replaced by tablet e8c4d793f10147fba5e4f44474575bf0) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.101095 1179 ts_tablet_manager.cc:1929] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:08.101256 1179 log.cc:1199] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/61ff3274c7104c73bd35768c229833b1
I20251029 08:52:08.101660 1179 ts_tablet_manager.cc:1950] T 61ff3274c7104c73bd35768c229833b1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:08.087303 1183 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.104521 1183 tablet_bootstrap.cc:654] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.105391 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 09759a50a82f4b758449da9fbbc1f72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.106220 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:08.112769 1176 ts_tablet_manager.cc:1916] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.114429 1179 tablet_replica.cc:333] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:08.114661 1179 raft_consensus.cc:2243] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.114841 1179 raft_consensus.cc:2272] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.114953 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.115430 1179 ts_tablet_manager.cc:1916] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.116951 1183 tablet_bootstrap.cc:492] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.117132 1183 ts_tablet_manager.cc:1403] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251029 08:52:08.120203 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 09759a50a82f4b758449da9fbbc1f72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.120949 1176 ts_tablet_manager.cc:1929] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:08.121096 1176 log.cc:1199] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:08.121592 1176 ts_tablet_manager.cc:1950] T a7fdbe6681894ffa8264cdfa486f16e6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.123993 1176 tablet_replica.cc:333] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.124332 1176 raft_consensus.cc:2243] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.124593 1176 raft_consensus.cc:2272] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.117985 1183 raft_consensus.cc:359] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.124959 1183 raft_consensus.cc:385] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.125098 1183 raft_consensus.cc:740] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.125669 1183 consensus_queue.cc:260] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.126282 1183 ts_tablet_manager.cc:1434] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20251029 08:52:08.126693 1183 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.128573 1438 raft_consensus.cc:493] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.128736 1438 raft_consensus.cc:515] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.129354 1438 leader_election.cc:290] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:08.122906 1178 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.131209 1178 ts_tablet_manager.cc:1403] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.180s user 0.003s sys 0.031s
I20251029 08:52:08.131996 1176 ts_tablet_manager.cc:1916] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.134457 890 tablet_service.cc:1505] Processing CreateTablet for tablet aafa846cb2144cec9ce4b28e61d91961 (DEFAULT_TABLE table=default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.134727 1183 tablet_bootstrap.cc:654] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.135022 897 tablet_service.cc:1505] Processing CreateTablet for tablet bbc4345cabbc476485b6a040a076ddaf (DEFAULT_TABLE table=default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5 [id=0e01efd35bcd43dc88e2d92f0734b958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.135411 902 tablet_service.cc:1505] Processing CreateTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 (DEFAULT_TABLE table=default.Table_32a7c0c8b359467995851e70b6448f62 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.135771 895 tablet_service.cc:1505] Processing CreateTablet for tablet de78ea608c904aeea7f7145b9c80f45b (DEFAULT_TABLE table=default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.136270 1433 raft_consensus.cc:493] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.136469 1433 raft_consensus.cc:515] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.136828 1178 raft_consensus.cc:359] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.137080 1178 raft_consensus.cc:385] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.137161 1178 raft_consensus.cc:740] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.137286 1433 leader_election.cc:290] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.137449 1433 raft_consensus.cc:493] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.137511 1178 consensus_queue.cc:260] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.137604 1433 raft_consensus.cc:515] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.138149 1433 leader_election.cc:290] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:08.138243 1178 ts_tablet_manager.cc:1434] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 08:52:08.138309 1353 raft_consensus.cc:493] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.138422 1353 raft_consensus.cc:515] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.138989 1353 leader_election.cc:290] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.139341 1179 ts_tablet_manager.cc:1929] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.139083 1178 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.139485 1179 log.cc:1199] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/933817bf2058464f837ba96921c4ed63
I20251029 08:52:08.139861 1179 ts_tablet_manager.cc:1950] T 933817bf2058464f837ba96921c4ed63 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:08.140247 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2947083e3e447d5addae5effd186eb8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.140447 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.140513 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a20b64ff177241309e5d59e84f0bafa2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.141494 1178 tablet_bootstrap.cc:654] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:08.142613 861 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.141858 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
W20251029 08:52:08.142865 861 leader_election.cc:343] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.142944 781 raft_consensus.cc:2468] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:08.143083 861 leader_election.cc:343] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.143275 861 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a2947083e3e447d5addae5effd186eb8
I20251029 08:52:08.143414 861 leader_election.cc:304] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:08.143986 1353 raft_consensus.cc:2749] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:08.144806 861 leader_election.cc:343] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:08.144972 861 leader_election.cc:304] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:08.145464 861 leader_election.cc:304] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:08.146231 1353 raft_consensus.cc:2749] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.146433 1353 raft_consensus.cc:2804] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:08.146534 1353 raft_consensus.cc:493] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:08.146620 1353 raft_consensus.cc:3060] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:08.160712 1461 raft_consensus.cc:493] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.160580 1176 ts_tablet_manager.cc:1929] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.161057 1461 raft_consensus.cc:515] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.161671 1176 log.cc:1199] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4cf906c73dc84403bcc3dece74a822bd
I20251029 08:52:08.161902 1461 leader_election.cc:290] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.163566 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09759a50a82f4b758449da9fbbc1f72d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:08.164371 720 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.164213 1176 ts_tablet_manager.cc:1950] T 4cf906c73dc84403bcc3dece74a822bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.166097 1183 tablet_bootstrap.cc:492] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.166285 1183 ts_tablet_manager.cc:1403] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.004s
I20251029 08:52:08.155782 1353 raft_consensus.cc:515] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.169437 1178 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.169811 1178 ts_tablet_manager.cc:1403] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.006s
I20251029 08:52:08.171772 1178 raft_consensus.cc:359] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.172290 1178 raft_consensus.cc:385] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.172878 1178 raft_consensus.cc:740] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.172986 1183 raft_consensus.cc:359] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.173218 1183 raft_consensus.cc:385] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.173332 1183 raft_consensus.cc:740] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.173592 1183 consensus_queue.cc:260] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.174163 1183 ts_tablet_manager.cc:1434] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251029 08:52:08.174535 1183 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.174031 1178 consensus_queue.cc:260] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.157536 1448 raft_consensus.cc:515] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.177125 1176 tablet_replica.cc:333] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.178030 1178 ts_tablet_manager.cc:1434] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251029 08:52:08.178964 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.178534 1176 raft_consensus.cc:2243] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.179670 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafa846cb2144cec9ce4b28e61d91961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.180187 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 9702a7d8b8304b49a250c129ecf28303 with delete_type TABLET_DATA_DELETED (replaced by tablet 7be4bcf9e3324bd884e946fae20648fd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.180538 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.187904 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ab9df6f768410eb4f66cffdcd68df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.189320 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de78ea608c904aeea7f7145b9c80f45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.191386 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc4345cabbc476485b6a040a076ddaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.193145 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 933817bf2058464f837ba96921c4ed63 with delete_type TABLET_DATA_DELETED (replaced by tablet e8c4d793f10147fba5e4f44474575bf0) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.193460 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ff3274c7104c73bd35768c229833b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c1a8a1e4cc14f1aa6e4f91174f45127) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.194550 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc04616301354cfe873c825a9f5d7ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.198911 1177 ts_tablet_manager.cc:1434] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.182s user 0.003s sys 0.000s
I20251029 08:52:08.200567 1177 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:08.201763 1176 raft_consensus.cc:2272] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.202225 1178 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.203648 1448 leader_election.cc:290] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.205921 1353 leader_election.cc:290] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:08.206638 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:08.207010 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 9702a7d8b8304b49a250c129ecf28303 with delete_type TABLET_DATA_DELETED (replaced by tablet 7be4bcf9e3324bd884e946fae20648fd) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.207046 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:08.207252 781 raft_consensus.cc:2468] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:08.207863 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.208209 784 raft_consensus.cc:3060] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:08.208663 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ff3274c7104c73bd35768c229833b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c1a8a1e4cc14f1aa6e4f91174f45127) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.211894 784 raft_consensus.cc:2468] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:08.213907 1176 ts_tablet_manager.cc:1916] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.216140 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 933817bf2058464f837ba96921c4ed63 (table default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:08.217263 1461 raft_consensus.cc:493] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.217444 1461 raft_consensus.cc:515] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.218181 1461 leader_election.cc:290] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.218931 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 933817bf2058464f837ba96921c4ed63 with delete_type TABLET_DATA_DELETED (replaced by tablet e8c4d793f10147fba5e4f44474575bf0) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.220508 861 leader_election.cc:304] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:08.221215 861 leader_election.cc:304] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:08.221662 1178 tablet_bootstrap.cc:654] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.227999 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed66c3fa3f848df905a2a31fd553474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.237028 1448 raft_consensus.cc:2804] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:08.237300 1448 raft_consensus.cc:493] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:08.239472 1461 raft_consensus.cc:493] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.239676 1461 raft_consensus.cc:515] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.239894 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:08.240454 1461 leader_election.cc:290] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.243044 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a7fdbe6681894ffa8264cdfa486f16e6 (table default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]) successfully deleted
I20251029 08:52:08.243276 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4cf906c73dc84403bcc3dece74a822bd (table default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]) successfully deleted
I20251029 08:52:08.244424 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09759a50a82f4b758449da9fbbc1f72d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:08.251549 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d30183814f5e4f2ab9d71fe493eb801c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30183814f5e4f2ab9d71fe493eb801c already in progress: creating tablet
W20251029 08:52:08.251885 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7c3e50018f414f4b88e0e5d90cb7c925 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c3e50018f414f4b88e0e5d90cb7c925 already in progress: creating tablet
I20251029 08:52:08.255460 1461 raft_consensus.cc:493] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.255677 1461 raft_consensus.cc:515] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.256374 1461 leader_election.cc:290] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.258200 901 tablet_service.cc:1505] Processing CreateTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d (DEFAULT_TABLE table=default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.263051 1448 raft_consensus.cc:3060] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:08.262806 891 tablet_service.cc:1505] Processing CreateTablet for tablet 69d27944f3c14a6eb8331a904d6faa40 (DEFAULT_TABLE table=default.Table_2ec426aafdb6470a85870622c71f3f64 [id=ad2b093021204f7eaf35330fd0675654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.265507 1353 raft_consensus.cc:2804] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:08.266109 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a57b1dbbdf854bc6a6e1b5dc88b63bff" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.267176 1461 raft_consensus.cc:493] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.267256 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b7853b85d24a04b8b11caaa4fe8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.267369 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "983a406aeed243f29ab322f94a0f5239" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.267374 1461 raft_consensus.cc:515] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.268096 1461 leader_election.cc:290] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.270866 1353 raft_consensus.cc:697] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:08.275312 1178 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.275502 1178 ts_tablet_manager.cc:1403] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.011s
I20251029 08:52:08.276702 1178 raft_consensus.cc:359] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.276899 1178 raft_consensus.cc:385] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.276957 1178 raft_consensus.cc:740] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.277153 1178 consensus_queue.cc:260] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.279624 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d27944f3c14a6eb8331a904d6faa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.282776 1178 ts_tablet_manager.cc:1434] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:08.289345 1176 ts_tablet_manager.cc:1929] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.289554 1176 log.cc:1199] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:08.289968 1176 ts_tablet_manager.cc:1950] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.290786 1176 tablet_replica.cc:333] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.291446 1353 consensus_queue.cc:237] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.294799 1178 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.296558 1178 tablet_bootstrap.cc:654] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.300515 1176 raft_consensus.cc:2243] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.300756 1176 raft_consensus.cc:2272] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.301738 1176 ts_tablet_manager.cc:1916] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.303737 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a756d6b5b3648e88cd2bd3400e51bed" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:08.304304 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.304914 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "983a406aeed243f29ab322f94a0f5239" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.307920 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e96cbdc4b7845b3823617a8e9a8c8b2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.308422 1046 tablet_service.cc:1505] Processing CreateTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d (DEFAULT_TABLE table=default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.308905 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b7853b85d24a04b8b11caaa4fe8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:08.312280 720 leader_election.cc:343] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.312816 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4deeec492a2545ca86583c6ea899a0ab" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.313274 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a57b1dbbdf854bc6a6e1b5dc88b63bff" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:08.313977 720 leader_election.cc:343] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.314230 720 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.314428 720 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.324892 1042 tablet_service.cc:1505] Processing CreateTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 (DEFAULT_TABLE table=default.Table_25a00ec5d08741a18f645f3d31d05ef5 [id=14f726f4d8104eaf99f5dbb58f9ce22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.325444 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af38662420d14ac1a7cb5fd03fc1b9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.330847 1176 ts_tablet_manager.cc:1929] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:08.332060 718 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.332255 748 tablet_service.cc:1505] Processing CreateTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 (DEFAULT_TABLE table=default.Table_25a00ec5d08741a18f645f3d31d05ef5 [id=14f726f4d8104eaf99f5dbb58f9ce22c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:08.336249 861 leader_election.cc:343] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.336745 861 leader_election.cc:343] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.341572 749 tablet_service.cc:1505] Processing CreateTablet for tablet 69d27944f3c14a6eb8331a904d6faa40 (DEFAULT_TABLE table=default.Table_2ec426aafdb6470a85870622c71f3f64 [id=ad2b093021204f7eaf35330fd0675654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.342155 1176 log.cc:1199] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:08.342756 1176 ts_tablet_manager.cc:1950] T 9702a7d8b8304b49a250c129ecf28303 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.346469 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af38662420d14ac1a7cb5fd03fc1b9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.346498 1178 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.346712 1178 ts_tablet_manager.cc:1403] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.002s
I20251029 08:52:08.348126 1178 raft_consensus.cc:359] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.348346 1178 raft_consensus.cc:385] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.348452 1178 raft_consensus.cc:740] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.348709 1178 consensus_queue.cc:260] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.349423 1178 ts_tablet_manager.cc:1434] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:08.349593 889 tablet_service.cc:1505] Processing CreateTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 (DEFAULT_TABLE table=default.Table_25a00ec5d08741a18f645f3d31d05ef5 [id=14f726f4d8104eaf99f5dbb58f9ce22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.349730 1178 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.353039 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d27944f3c14a6eb8331a904d6faa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.354105 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:08.354858 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2e96cbdc4b7845b3823617a8e9a8c8b2
I20251029 08:52:08.354979 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2ec426aafdb6470a85870622c71f3f64" } new_table_name: "default.Table_465b5a50b6e54ff686adb35a53a61809" modify_external_catalogs: true
I20251029 08:52:08.355643 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3fd4fbe8a2841d097f0f0d235f27e04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.356768 1178 tablet_bootstrap.cc:654] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.357030 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a57acf768e244dea851580de4ce3b4d3" } new_table_name: "default.Table_d6dc6ba680874a51afe94c07adb83e2b" modify_external_catalogs: true
I20251029 08:52:08.357585 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3dc21093f304d13933e68c784dbfe5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.358197 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3dc21093f304d13933e68c784dbfe5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:08.360206 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3fd4fbe8a2841d097f0f0d235f27e04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.369112 1176 tablet_replica.cc:333] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.369432 1176 raft_consensus.cc:2243] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.369721 1176 raft_consensus.cc:2272] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.372068 1176 ts_tablet_manager.cc:1916] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.377511 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.382436 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af38662420d14ac1a7cb5fd03fc1b9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.384080 718 leader_election.cc:304] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.384114 1461 raft_consensus.cc:493] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.385274 1461 raft_consensus.cc:515] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.387992 1461 leader_election.cc:290] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:08.388269 718 raft_consensus.cc:2686] T 09759a50a82f4b758449da9fbbc1f72d P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:08.395370 1183 tablet_bootstrap.cc:654] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.396806 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 4deeec492a2545ca86583c6ea899a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.397634 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39dc5ebeb36043e0aa5955bacd06c2ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.398316 1572 mvcc.cc:204] Tried to move back new op lower bound from 7216037588539908096 to 7216037585450450944. Current Snapshot: MvccSnapshot[applied={T|T < 7216037588539908096}]
W20251029 08:52:08.400085 720 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.403122 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 1dd62833c9d548adabbbf71856a4b181 (DEFAULT_TABLE table=default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.403630 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd62833c9d548adabbbf71856a4b181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.405182 1600 tablet.cc:1722] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:08.413030 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5051ff776e4975b917db0393a1f945 (DEFAULT_TABLE table=default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.413677 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5051ff776e4975b917db0393a1f945. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.417691 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 4deeec492a2545ca86583c6ea899a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.418092 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:08.420281 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 67407.9us
I20251029 08:52:08.436245 590 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet cd6208b92fe94dc881cf45b4627d68f2
I20251029 08:52:08.436859 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2c760e56e16a422eb0666aa83818633f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.437777 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c760e56e16a422eb0666aa83818633f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.438426 1600 tablet.cc:1722] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:08.442531 902 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5051ff776e4975b917db0393a1f945 (DEFAULT_TABLE table=default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.443022 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5051ff776e4975b917db0393a1f945. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:08.447054 718 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.448401 718 leader_election.cc:304] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.448863 1366 raft_consensus.cc:2749] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:08.449023 718 leader_election.cc:343] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.449168 718 leader_election.cc:304] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.449625 1366 raft_consensus.cc:2749] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.450079 1366 raft_consensus.cc:493] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.450201 1366 raft_consensus.cc:515] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.450500 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.450757 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.450834 1366 leader_election.cc:290] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.450978 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.451390 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4deeec492a2545ca86583c6ea899a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.451861 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b534aaa51b4fad9dc3fc915de947f1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.451390 750 tablet_service.cc:1505] Processing CreateTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d (DEFAULT_TABLE table=default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.452874 1461 raft_consensus.cc:493] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.453613 754 tablet_service.cc:1505] Processing CreateTablet for tablet 1dd62833c9d548adabbbf71856a4b181 (DEFAULT_TABLE table=default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:08.453598 720 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.453581 759 tablet_service.cc:1505] Processing CreateTablet for tablet 7b5051ff776e4975b917db0393a1f945 (DEFAULT_TABLE table=default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.455098 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.455322 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd62833c9d548adabbbf71856a4b181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.455600 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5051ff776e4975b917db0393a1f945. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.454764 1461 raft_consensus.cc:515] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.460898 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b7853b85d24a04b8b11caaa4fe8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.462757 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64f73597d1764540bcaf472d3e890ee8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.462997 1176 ts_tablet_manager.cc:1929] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.463724 1176 log.cc:1199] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7792ee223dbe43aa82990541caced045
I20251029 08:52:08.463747 1461 leader_election.cc:290] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.464535 1176 ts_tablet_manager.cc:1950] T 7792ee223dbe43aa82990541caced045 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.464964 1176 tablet_replica.cc:333] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.464421 1178 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.466461 1177 tablet_bootstrap.cc:654] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.466568 1176 raft_consensus.cc:2243] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:08.466383 720 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.466902 1176 raft_consensus.cc:2272] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:08.467651 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 already in progress: creating tablet
W20251029 08:52:08.468007 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4deeec492a2545ca86583c6ea899a0ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4deeec492a2545ca86583c6ea899a0ab already in progress: creating tablet
I20251029 08:52:08.469441 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 09759a50a82f4b758449da9fbbc1f72d (table default.Table_0f93256544fb4b56adbbcaf8c50ecb42 [id=dc9c30acd58449e4bcb9217b628b59dc]) successfully deleted
W20251029 08:52:08.469550 1568 log.cc:927] Time spent T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251029 08:52:08.469799 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9702a7d8b8304b49a250c129ecf28303 (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
I20251029 08:52:08.469882 1176 ts_tablet_manager.cc:1916] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.470075 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7792ee223dbe43aa82990541caced045 (table default.Table_9b80aae43bb845d1bb98687e0e6116f6 [id=335946e15b7c424ba3b769c46bd76ee9]) successfully deleted
I20251029 08:52:08.473421 1178 ts_tablet_manager.cc:1403] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.015s
I20251029 08:52:08.474172 1178 raft_consensus.cc:359] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.482117 1178 raft_consensus.cc:385] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.483146 1178 raft_consensus.cc:740] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.484236 1178 consensus_queue.cc:260] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.483769 901 tablet_service.cc:1505] Processing CreateTablet for tablet 1dd62833c9d548adabbbf71856a4b181 (DEFAULT_TABLE table=default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:08.485674 1178 ts_tablet_manager.cc:1434] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251029 08:52:08.485896 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 2 got 0
I20251029 08:52:08.487116 1178 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.489627 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 2 got 0
I20251029 08:52:08.489915 589 catalog_manager.cc:5649] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:08.490270 1176 ts_tablet_manager.cc:1929] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:08.490406 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:08.491698 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd62833c9d548adabbbf71856a4b181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:08.497320 1178 tablet_bootstrap.cc:654] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.508260 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b55ae84c8d441459a5931abda471f6c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.508683 1448 raft_consensus.cc:515] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.513718 1448 leader_election.cc:290] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.517761 1461 raft_consensus.cc:493] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.517870 996 leader_election.cc:304] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.518422 1438 raft_consensus.cc:2749] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.518747 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ed199b700f4f92858c4edb95217d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.519025 1179 tablet_replica.cc:333] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:08.519212 1179 raft_consensus.cc:2243] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.519387 1179 raft_consensus.cc:2272] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.519984 1179 ts_tablet_manager.cc:1916] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:08.521063 1355 kernel_stack_watchdog.cc:198] Thread 1262 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 311ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1cea79b980 (unknown) at ??:0
@ 0x93d797 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a2e4d __interceptor_pthread_create at ??:?
@ 0x7f1ce7184be1 kudu::Thread::StartThread() at ??:0
@ 0xa75107 kudu::Thread::Create() at ??:?
@ 0x7f1cee101293 kudu::rpc::ServicePool::Init() at ??:0
@ 0x7f1cf05aad91 kudu::RpcServer::RegisterService() at ??:0
@ 0x7f1cf05cc1a8 kudu::server::ServerBase::RegisterService() at ??:0
@ 0x7f1cf05cdf0c kudu::server::ServerBase::Start() at ??:0
@ 0x7f1cf1bae85c kudu::kserver::KuduServer::Start() at ??:0
@ 0x7f1cfadf6d1f kudu::master::Master::StartAsync() at ??:0
@ 0x7f1cfadf60b7 kudu::master::Master::Start() at ??:0
@ 0x7f1cfae58afb kudu::master::RunMasterServer() at ??:0
@ 0xaa1877 kudu::tools::(anonymous namespace)::MasterRun() at ??:?
@ 0x9bc2b1 std::_Function_handler<>::_M_invoke() at ??:?
@ 0x7f1cfb7fc159 std::function<>::operator()() at ??:0
I20251029 08:52:08.527292 1179 ts_tablet_manager.cc:1929] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.527412 1179 log.cc:1199] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/24ed199b700f4f92858c4edb95217d03
I20251029 08:52:08.527771 1179 ts_tablet_manager.cc:1950] T 24ed199b700f4f92858c4edb95217d03 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:08.528944 1262 thread.cc:641] rpc worker (service pool kudu.server.GenericService) Time spent creating pthread: real 0.608s user 0.080s sys 0.097s
W20251029 08:52:08.529072 1262 thread.cc:608] rpc worker (service pool kudu.server.GenericService) Time spent starting thread: real 0.609s user 0.080s sys 0.097s
I20251029 08:52:08.530905 1175 ts_tablet_manager.cc:1929] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.533330 1176 log.cc:1199] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/61ff3274c7104c73bd35768c229833b1
I20251029 08:52:08.533350 1461 raft_consensus.cc:515] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.533753 1175 log.cc:1199] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e72f64830e924fa48e2fa4cf81ccbba3
I20251029 08:52:08.533898 1176 ts_tablet_manager.cc:1950] T 61ff3274c7104c73bd35768c229833b1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.534196 1461 leader_election.cc:290] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.534981 1176 tablet_replica.cc:333] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.535006 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 24ed199b700f4f92858c4edb95217d03 (table default.Table_66c8696f5f854516b099c64d58891cdb [id=4106780aaf1545828d89da1558d8f58d]) successfully deleted
I20251029 08:52:08.535089 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.535183 1176 raft_consensus.cc:2243] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.535255 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 61ff3274c7104c73bd35768c229833b1 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:08.535367 1176 raft_consensus.cc:2272] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:08.535601 720 leader_election.cc:343] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.536861 1183 tablet_bootstrap.cc:492] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.537035 1183 ts_tablet_manager.cc:1403] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.363s user 0.002s sys 0.002s
I20251029 08:52:08.537986 1183 raft_consensus.cc:359] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.538232 1183 raft_consensus.cc:385] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.538311 1183 raft_consensus.cc:740] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.538646 1183 consensus_queue.cc:260] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.539958 1183 ts_tablet_manager.cc:1434] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20251029 08:52:08.540287 1183 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.542541 1183 tablet_bootstrap.cc:654] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.544935 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0622352c737244ae90d1a8ac1a960864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.545603 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0622352c737244ae90d1a8ac1a960864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.547199 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ae561a995c94fadb288b6395c55a1bf from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:08.548489 1176 ts_tablet_manager.cc:1916] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:08.553407 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69d27944f3c14a6eb8331a904d6faa40 consensus state has no leader
I20251029 08:52:08.553793 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_af1eb2fb69584b62b01d7ec5d5711afd" } new_table_name: "default.Table_04d72f6fb117444a96f5d646217085e2" modify_external_catalogs: true
W20251029 08:52:08.558753 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee8117e04f5f40e98ea755dc461067dd consensus state has no leader
I20251029 08:52:08.559213 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2e2db23cf2c6466b9faa928d7adac713" } new_table_name: "default.Table_13d81683d3304abba1837078e6685abd" modify_external_catalogs: true
I20251029 08:52:08.560923 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b9370f9fc0ee4f829cbd096420cf8c53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.561457 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9370f9fc0ee4f829cbd096420cf8c53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.569247 1178 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.569278 1175 ts_tablet_manager.cc:1950] T e72f64830e924fa48e2fa4cf81ccbba3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:08.569420 1178 ts_tablet_manager.cc:1403] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.011s
I20251029 08:52:08.569909 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 288) took 1343 ms. Trace:
I20251029 08:52:08.570008 1175 rpcz_store.cc:276] 1029 08:52:07.225915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.226507 (+ 592us) service_pool.cc:225] Handling call
1029 08:52:07.917350 (+690843us) ts_tablet_manager.cc:1151] Deleting tablet e72f64830e924fa48e2fa4cf81ccbba3
1029 08:52:07.917356 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.238450 (+321094us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.517465 (+279015us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.569645 (+ 52180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":690651}
I20251029 08:52:08.570732 1175 tablet_replica.cc:333] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:08.570773 1178 raft_consensus.cc:359] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.570955 1178 raft_consensus.cc:385] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.570948 1175 raft_consensus.cc:2243] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.571039 1178 raft_consensus.cc:740] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.571252 1178 consensus_queue.cc:260] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.571808 1178 ts_tablet_manager.cc:1434] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:08.571805 1175 raft_consensus.cc:2272] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.572396 1175 ts_tablet_manager.cc:1916] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.575783 1438 raft_consensus.cc:493] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.575959 1438 raft_consensus.cc:515] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.576630 1438 leader_election.cc:290] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:08.577751 998 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:08.584784 1178 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.587111 1178 tablet_bootstrap.cc:654] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:08.589035 996 leader_election.cc:343] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:08.589179 996 leader_election.cc:304] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.589615 1438 raft_consensus.cc:2749] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.589690 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630] alter complete (version 5)
I20251029 08:52:08.590544 1176 ts_tablet_manager.cc:1929] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:08.590529 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c778ace3a094ba7a37d7def4c6497a7" } new_table_name: "default.Table_a5e58cbcddc94f118f224404b14c6790" modify_external_catalogs: true
I20251029 08:52:08.590657 1176 log.cc:1199] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/933817bf2058464f837ba96921c4ed63
W20251029 08:52:08.590641 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 130245us
I20251029 08:52:08.591053 1176 ts_tablet_manager.cc:1950] T 933817bf2058464f837ba96921c4ed63 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.592576 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_18a7eae91e5b4ab595c36f630248860f" } modify_external_catalogs: true
I20251029 08:52:08.592747 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_18a7eae91e5b4ab595c36f630248860f" } modify_external_catalogs: true
I20251029 08:52:08.594992 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 933817bf2058464f837ba96921c4ed63 (table default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:08.596648 1176 tablet_replica.cc:333] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.596943 1176 raft_consensus.cc:2243] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.597138 1176 raft_consensus.cc:2272] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.597774 1176 ts_tablet_manager.cc:1916] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.602380 1178 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.602560 1178 ts_tablet_manager.cc:1403] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20251029 08:52:08.603508 1178 raft_consensus.cc:359] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.603724 1178 raft_consensus.cc:385] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.603785 1178 raft_consensus.cc:740] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.604404 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4deeec492a2545ca86583c6ea899a0ab" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.604887 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e96cbdc4b7845b3823617a8e9a8c8b2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.605284 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39dc5ebeb36043e0aa5955bacd06c2ef" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.605675 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b534aaa51b4fad9dc3fc915de947f1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.606235 1605 tablet.cc:1722] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:08.606864 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64f73597d1764540bcaf472d3e890ee8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.607255 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.607662 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e72f64830e924fa48e2fa4cf81ccbba3 (table default.Table_4ad4591d3d494b20a6f4f88479c97229 [id=32b517f85ee64480877ae86fe3da3599]) successfully deleted
W20251029 08:52:08.608541 718 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.608690 718 leader_election.cc:304] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.608955 1461 raft_consensus.cc:2749] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:08.609120 718 leader_election.cc:343] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.609230 718 leader_election.cc:304] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:08.609380 718 raft_consensus.cc:2686] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:08.609615 718 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.609689 718 leader_election.cc:304] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:08.609982 718 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.610059 718 leader_election.cc:304] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:08.610332 718 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.610407 718 leader_election.cc:304] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.610747 1366 raft_consensus.cc:2749] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.610903 1366 raft_consensus.cc:2749] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.611013 1366 raft_consensus.cc:2749] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.620456 1178 consensus_queue.cc:260] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.622740 1178 ts_tablet_manager.cc:1434] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20251029 08:52:08.623080 1178 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.624222 590 catalog_manager.cc:5649] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:08.624652 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_7b45e52828bf4727815bfbb7c1ddb9c3 [id=63588e39919a481c856255fd848bbf98]). Expected version 1 got 0
I20251029 08:52:08.627945 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:08.628206 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 2 got 0
I20251029 08:52:08.629336 1262 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40435
I20251029 08:52:08.630980 1617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40435 every 8 connection(s)
I20251029 08:52:08.632665 1178 tablet_bootstrap.cc:654] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.637748 1262 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
I20251029 08:52:08.641197 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_770bebdc34fe40eab00f06e1d53a4d06" } modify_external_catalogs: true
I20251029 08:52:08.641372 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_770bebdc34fe40eab00f06e1d53a4d06" } modify_external_catalogs: true
I20251029 08:52:08.642940 1176 ts_tablet_manager.cc:1929] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.643071 1176 log.cc:1199] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2e96cbdc4b7845b3823617a8e9a8c8b2
I20251029 08:52:08.643399 1176 ts_tablet_manager.cc:1950] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.648249 1176 tablet_replica.cc:333] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.648442 1176 raft_consensus.cc:2243] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.648617 1176 raft_consensus.cc:2272] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.658524 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:08.658699 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:08.658772 783 raft_consensus.cc:3060] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:08.660843 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 1262
I20251029 08:52:08.660404 1068 raft_consensus.cc:3060] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:08.663597 783 raft_consensus.cc:2468] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:08.664781 1178 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:08.664954 1178 ts_tablet_manager.cc:1403] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20251029 08:52:08.665982 1178 raft_consensus.cc:359] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.666181 1178 raft_consensus.cc:385] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.666289 1178 raft_consensus.cc:740] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.666507 1178 consensus_queue.cc:260] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.667187 1178 ts_tablet_manager.cc:1434] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:08.667507 1178 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:08.668061 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 1ae561a995c94fadb288b6395c55a1bf
I20251029 08:52:08.668599 594 catalog_manager.cc:6719] ReplaceTablet: tablet 1ae561a995c94fadb288b6395c55a1bf deleted and replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54
I20251029 08:52:08.668884 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 39dc5ebeb36043e0aa5955bacd06c2ef from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:08.669394 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3137cb7a7def4c5693f750cec7eb3f47" } modify_external_catalogs: true
I20251029 08:52:08.669588 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3137cb7a7def4c5693f750cec7eb3f47" } modify_external_catalogs: true
I20251029 08:52:08.670794 1178 tablet_bootstrap.cc:654] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.671903 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.672556 1068 raft_consensus.cc:1275] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 08:52:08.672920 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 1ae561a995c94fadb288b6395c55a1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae561a995c94fadb288b6395c55a1bf already in progress: creating tablet
I20251029 08:52:08.673949 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.674459 1183 tablet_bootstrap.cc:492] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.674593 1183 ts_tablet_manager.cc:1403] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.000s
I20251029 08:52:08.675385 1183 raft_consensus.cc:359] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.675567 1183 raft_consensus.cc:385] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.675664 1183 raft_consensus.cc:740] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.675891 1183 consensus_queue.cc:260] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.676519 1183 ts_tablet_manager.cc:1434] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:08.676867 1183 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.677837 1461 raft_consensus.cc:493] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.678009 1461 raft_consensus.cc:515] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.678665 1461 leader_election.cc:290] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.680502 1438 raft_consensus.cc:493] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.680656 1438 raft_consensus.cc:515] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.681263 1438 leader_election.cc:290] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:08.683132 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 351047c725f8488abbd074bd40790b48 consensus state has no leader
I20251029 08:52:08.683684 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:08.684257 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea13e9ea2c11479ea86e3fd7f2a9e59c consensus state has no leader
I20251029 08:52:08.684953 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 215cd666c71a4179a19e298868a88e96 from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:08.685365 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2773cae9814df79ee60ea8b0fe4e9d consensus state has no leader
I20251029 08:52:08.685509 1183 tablet_bootstrap.cc:654] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.690618 1175 ts_tablet_manager.cc:1929] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.690765 1175 log.cc:1199] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:08.694067 1176 ts_tablet_manager.cc:1916] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.691143 1175 ts_tablet_manager.cc:1950] T a20b64ff177241309e5d59e84f0bafa2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:08.695495 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 290) took 1438 ms. Trace:
I20251029 08:52:08.695626 1175 rpcz_store.cc:276] 1029 08:52:07.256974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.257030 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:08.570308 (+1313278us) ts_tablet_manager.cc:1151] Deleting tablet a20b64ff177241309e5d59e84f0bafa2
1029 08:52:08.570320 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.581000 (+ 10680us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.690450 (+109450us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.695456 (+ 5006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1313083}
I20251029 08:52:08.696142 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a20b64ff177241309e5d59e84f0bafa2 (table default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]) successfully deleted
I20251029 08:52:08.696597 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 291) took 1400 ms. Trace:
I20251029 08:52:08.696722 1175 rpcz_store.cc:276] 1029 08:52:07.296561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.296706 (+ 145us) service_pool.cc:225] Handling call
1029 08:52:08.696430 (+1399724us) ts_tablet_manager.cc:1151] Deleting tablet ef0a7b31ced24388bc9d9964a42fa203
1029 08:52:08.696442 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.696556 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1399504}
I20251029 08:52:08.696815 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9165c4bf484a48b4a0b7246cb11571bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:08.697203 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ef0a7b31ced24388bc9d9964a42fa203 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0a7b31ced24388bc9d9964a42fa203 already in progress: creating tablet
W20251029 08:52:08.697414 720 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.697669 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 296) took 1267 ms. Trace:
I20251029 08:52:08.697781 1175 rpcz_store.cc:276] 1029 08:52:07.430465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.430516 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:08.697531 (+1267015us) ts_tablet_manager.cc:1151] Deleting tablet 23b534aaa51b4fad9dc3fc915de947f1
1029 08:52:08.697541 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.697634 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266757}
W20251029 08:52:08.698186 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 23b534aaa51b4fad9dc3fc915de947f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b534aaa51b4fad9dc3fc915de947f1 already in progress: creating tablet
I20251029 08:52:08.699497 783 raft_consensus.cc:1275] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 08:52:08.700480 861 leader_election.cc:343] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.700781 1353 consensus_queue.cc:1048] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:08.705330 1461 raft_consensus.cc:493] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.705492 1461 raft_consensus.cc:515] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.706029 1461 leader_election.cc:290] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.706621 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55842c18dbef4513a61dd9629f2e5a4c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:08.707054 720 leader_election.cc:343] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.711504 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_565b58db3efc46a9bd80cb9ae264bab1" } modify_external_catalogs: true
I20251029 08:52:08.711704 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_565b58db3efc46a9bd80cb9ae264bab1" } modify_external_catalogs: true
I20251029 08:52:08.712136 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e4f80728a7a04583bada485707beb806
I20251029 08:52:08.712839 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 4e2773cae9814df79ee60ea8b0fe4e9d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:08.713269 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5d9918fbcc644b93bee6bc87ed741463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.713855 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d9918fbcc644b93bee6bc87ed741463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.715224 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d5036ccd9a194fcf828317d2b8060b73" } modify_external_catalogs: true
I20251029 08:52:08.715541 861 leader_election.cc:304] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:08.715756 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d5036ccd9a194fcf828317d2b8060b73" } modify_external_catalogs: true
I20251029 08:52:08.715818 1353 raft_consensus.cc:2804] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:08.715950 1353 raft_consensus.cc:697] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:08.716293 1353 consensus_queue.cc:237] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.717289 901 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.717769 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:08.718700 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e4f80728a7a04583bada485707beb806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4f80728a7a04583bada485707beb806 already in progress: creating tablet
I20251029 08:52:08.723707 1601 consensus_queue.cc:1048] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:08.728381 1177 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:08.729543 1177 ts_tablet_manager.cc:1403] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.529s user 0.006s sys 0.014s
W20251029 08:52:08.735499 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 125134us
W20251029 08:52:08.735802 718 leader_election.cc:343] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.735952 718 leader_election.cc:304] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.737435 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (table default.Table_b9b1c4a272214856ab50337015cdbd7b [id=4856bf7f477c47d58060d1728c544aa0]) successfully deleted
I20251029 08:52:08.738238 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9165c4bf484a48b4a0b7246cb11571bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.780098 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c3e50018f414f4b88e0e5d90cb7c925 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:08.780313 593 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0] alter complete (version 2)
I20251029 08:52:08.746456 1183 tablet_bootstrap.cc:492] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:08.932271 1183 ts_tablet_manager.cc:1403] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.255s user 0.002s sys 0.008s
I20251029 08:52:08.767406 1619 sys_catalog.cc:263] Verifying existing consensus state
I20251029 08:52:08.741035 1461 raft_consensus.cc:2749] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.738408 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55842c18dbef4513a61dd9629f2e5a4c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:08.777428 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:08.813581 1627 raft_consensus.cc:493] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.933614 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1d27485babbd4ce5afd764a5d187d095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.933670 1627 raft_consensus.cc:515] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:08.741170 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8cb0c6c6a1c4877a89476d4822f48db" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:08.741492 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:08.741729 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:08.934267 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d27485babbd4ce5afd764a5d187d095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:08.744740 718 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.934458 718 leader_election.cc:304] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.934610 1627 leader_election.cc:290] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:08.776963 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:08.780690 1178 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
W20251029 08:52:08.934828 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 192968us
I20251029 08:52:08.935052 1178 ts_tablet_manager.cc:1403] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.268s user 0.005s sys 0.008s
I20251029 08:52:08.780611 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:08.785065 1176 ts_tablet_manager.cc:1929] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.785012 1641 tablet.cc:1722] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:08.935560 1176 log.cc:1199] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:08.841501 1616 raft_consensus.cc:493] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.936046 1176 ts_tablet_manager.cc:1950] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:08.935958 1178 raft_consensus.cc:359] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.936103 1616 raft_consensus.cc:515] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.936199 1178 raft_consensus.cc:385] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.936308 1178 raft_consensus.cc:740] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.936228 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4ab4c150f8dd4dfdba46c3555d68e653"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.936563 1176 tablet_replica.cc:333] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:08.936681 1461 raft_consensus.cc:493] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.936800 1176 raft_consensus.cc:2243] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:08.936828 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ab4c150f8dd4dfdba46c3555d68e653 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.936827 1461 raft_consensus.cc:515] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.936980 1176 raft_consensus.cc:2272] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.935554 1366 raft_consensus.cc:2749] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:08.937196 1366 raft_consensus.cc:493] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:08.937335 1366 raft_consensus.cc:515] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.937438 1461 leader_election.cc:290] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:08.937595 1176 ts_tablet_manager.cc:1916] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.746870 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 297) took 1316 ms. Trace:
I20251029 08:52:08.937861 1366 leader_election.cc:290] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:08.938038 998 leader_election.cc:304] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:08.937991 1175 rpcz_store.cc:276] 1029 08:52:07.430584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.430810 (+ 226us) service_pool.cc:225] Handling call
1029 08:52:08.742730 (+1311920us) ts_tablet_manager.cc:1151] Deleting tablet 363fcf8a3e0f4770a9743fddffa6495f
1029 08:52:08.742743 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.746825 (+ 4082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311737}
I20251029 08:52:08.938525 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 298) took 1507 ms. Trace:
I20251029 08:52:08.938644 1175 rpcz_store.cc:276] 1029 08:52:07.430750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.431039 (+ 289us) service_pool.cc:225] Handling call
1029 08:52:08.938347 (+1507308us) ts_tablet_manager.cc:1151] Deleting tablet 0ff4e625abf14fbd94a8102dfa5fc993
1029 08:52:08.938359 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.938487 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507126}
I20251029 08:52:08.938746 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_06f02760c1014a1dbe9f6eea969646a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.938922 1175 tablet_replica.cc:333] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:08.939126 1175 raft_consensus.cc:2243] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.939338 1175 raft_consensus.cc:2272] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:08.939334 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06f02760c1014a1dbe9f6eea969646a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.939965 1175 ts_tablet_manager.cc:1916] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.780872 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a3ba512bd4f04bd9aceb4cb210fe9689
W20251029 08:52:08.942443 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 363fcf8a3e0f4770a9743fddffa6495f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 363fcf8a3e0f4770a9743fddffa6495f already in progress: creating tablet
W20251029 08:52:08.942689 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0ff4e625abf14fbd94a8102dfa5fc993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ff4e625abf14fbd94a8102dfa5fc993 already in progress: creating tablet
I20251029 08:52:08.942901 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5820689813104d31a2a4a5bcf6d6c53a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.943171 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e67ddf5ceb148dfad8f7be9f26cff64" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:08.943465 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5820689813104d31a2a4a5bcf6d6c53a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.943450 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e808bff7558472099587a0f95c002ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:03 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.943675 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.797603 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:08.944168 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet b273c488f33c498890f1739380b18239 with delete_type TABLET_DATA_DELETED (replaced by tablet 29adc6d53af5419dbda1d41d75d37e71) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.944453 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b459b366cb402db2e7e8fee2b3e855 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c515944ecf4228a7cca63cfd3565d8) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.803670 596 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet bb554c191f0b4638a1abd123991c9ab0
I20251029 08:52:08.747530 1177 raft_consensus.cc:359] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.944677 1177 raft_consensus.cc:385] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.944775 1177 raft_consensus.cc:740] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.944797 1183 raft_consensus.cc:359] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.944976 1183 raft_consensus.cc:385] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:08.945051 1183 raft_consensus.cc:740] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:08.945003 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_25a00ec5d08741a18f645f3d31d05ef5" } new_table_name: "default.Table_69a638baaadc493f9439102862bb7607" modify_external_catalogs: true
I20251029 08:52:08.945026 1177 consensus_queue.cc:260] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.945286 1183 consensus_queue.cc:260] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:08.945571 1177 ts_tablet_manager.cc:1434] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.215s user 0.002s sys 0.000s
I20251029 08:52:08.945744 1183 ts_tablet_manager.cc:1434] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
W20251029 08:52:08.755784 861 consensus_peers.cc:597] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 933817bf2058464f837ba96921c4ed63. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:08.945953 1177 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:08.945919 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 190142us
I20251029 08:52:08.816769 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4e2773cae9814df79ee60ea8b0fe4e9d
I20251029 08:52:08.946835 1438 raft_consensus.cc:2749] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 08:52:08.946909 591 catalog_manager.cc:6719] ReplaceTablet: tablet 4e2773cae9814df79ee60ea8b0fe4e9d deleted and replaced by tablet 9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:08.784572 901 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.947227 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e0316d58fa5c49d0a294cd9a37632924" } new_table_name: "default.Table_04ec7a02b4e44860b339d5e95468a084" modify_external_catalogs: true
I20251029 08:52:08.947819 1177 tablet_bootstrap.cc:654] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.827786 588 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b799fa10ee554524b52cd04eead89ffc
I20251029 08:52:08.948649 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_98254c96ab3c40b6a58d2295fedd0d16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:08.949122 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b273c488f33c498890f1739380b18239" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:08.949229 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98254c96ab3c40b6a58d2295fedd0d16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.784794 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.950107 1605 tablet.cc:1722] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:08.784904 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:08.951012 589 catalog_manager.cc:5649] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:08.951455 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:08.952572 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63b08840d0fa40428c61a05b277ad750" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:08.953339 1179 tablet_replica.cc:333] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:08.877676 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (table default.Table_13d81683d3304abba1837078e6685abd [id=994af8859eb748c4a3ffee07d64d1e33], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ea13e9ea2c11479ea86e3fd7f2a9e59c consensus state has no leader
I20251029 08:52:08.953591 1179 raft_consensus.cc:2243] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.953781 1179 raft_consensus.cc:2272] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.954540 1179 ts_tablet_manager.cc:1916] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.957109 781 raft_consensus.cc:1275] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:08.964881 1588 consensus_queue.cc:1048] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:08.966286 1641 tablet.cc:1722] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:08.936892 1616 leader_election.cc:290] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:08.967274 996 leader_election.cc:343] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: Tablet not RUNNING: STOPPED
I20251029 08:52:08.967882 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.968158 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.968259 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.968264 1175 ts_tablet_manager.cc:1929] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.968771 1175 log.cc:1199] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/b273c488f33c498890f1739380b18239
I20251029 08:52:08.970098 1175 ts_tablet_manager.cc:1950] T b273c488f33c498890f1739380b18239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:08.970403 902 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.970660 901 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:08.971165 720 leader_election.cc:343] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.971411 720 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:08.971616 720 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:08.972779 1600 tablet.cc:1722] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 08:52:08.973829 861 consensus_peers.cc:597] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:08.975306 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 301) took 1522 ms. Trace:
I20251029 08:52:08.975708 1175 rpcz_store.cc:276] 1029 08:52:07.453010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.453330 (+ 320us) service_pool.cc:225] Handling call
1029 08:52:08.938865 (+1485535us) ts_tablet_manager.cc:1151] Deleting tablet b273c488f33c498890f1739380b18239
1029 08:52:08.938873 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.957275 (+ 18402us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.967845 (+ 10570us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.975061 (+ 7216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1485319}
I20251029 08:52:08.977324 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 300) took 1524 ms. Trace:
I20251029 08:52:08.977705 1175 rpcz_store.cc:276] 1029 08:52:07.452866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.453412 (+ 546us) service_pool.cc:225] Handling call
1029 08:52:08.976716 (+1523304us) ts_tablet_manager.cc:1151] Deleting tablet 24ed199b700f4f92858c4edb95217d03
1029 08:52:08.976727 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.977179 (+ 452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1523102}
I20251029 08:52:08.979086 1175 tablet_replica.cc:333] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:08.936563 1178 consensus_queue.cc:260] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:08.980369 1175 raft_consensus.cc:2243] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:08.980571 1175 raft_consensus.cc:2272] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:08.980713 1178 ts_tablet_manager.cc:1434] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251029 08:52:08.981160 1175 ts_tablet_manager.cc:1916] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:08.982421 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:08.983450 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet b273c488f33c498890f1739380b18239 (table default.Table_dc9e5c781bc0463d9df56e2db965b1bb [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
W20251029 08:52:08.983811 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 24ed199b700f4f92858c4edb95217d03 because the tablet was not found. No further retry: Not found: Tablet not found: 24ed199b700f4f92858c4edb95217d03
I20251029 08:52:08.987478 1619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap starting.
I20251029 08:52:08.987708 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]). Expected version 5 got 0
I20251029 08:52:08.987946 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 2 got 0
I20251029 08:52:08.988539 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8360e68864e74c0b8acca762339062b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.989171 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8360e68864e74c0b8acca762339062b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.989620 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6d54c5b6b6f84dc896c7cc341921076e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:08.991550 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d54c5b6b6f84dc896c7cc341921076e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:08.993028 1183 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:08.994982 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:08.995949 1183 tablet_bootstrap.cc:654] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:08.998888 1175 ts_tablet_manager.cc:1929] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.999047 1175 log.cc:1199] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bb6b86f39c994ea5930a6a970830093f
I20251029 08:52:08.999470 1175 ts_tablet_manager.cc:1950] T bb6b86f39c994ea5930a6a970830093f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:08.999694 1176 ts_tablet_manager.cc:1929] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:08.999840 1176 log.cc:1199] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/64f73597d1764540bcaf472d3e890ee8
W20251029 08:52:09.013104 720 consensus_peers.cc:597] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:09.025117 1600 tablet.cc:1717] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:09.026468 1071 raft_consensus.cc:3060] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:09.026439 1619 log.cc:826] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Log is configured to *not* fsync() on all Append() calls
W20251029 08:52:09.044422 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af38662420d14ac1a7cb5fd03fc1b9d8 consensus state has no leader
I20251029 08:52:09.045051 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.045832 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e1f5ac232fb544b3a9e3397ef1472c07" } new_table_name: "default.Table_3d67d4e4f61646fe8e8864734badbc5d" modify_external_catalogs: true
W20251029 08:52:09.046051 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:09.047334 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63dfcd944c1f487b8911ae383770d930 consensus state has no leader
I20251029 08:52:09.047756 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7b45e52828bf4727815bfbb7c1ddb9c3" } new_table_name: "default.Table_cbc899e7cb9244738c40f175d924ad40" modify_external_catalogs: true
I20251029 08:52:09.049969 1627 raft_consensus.cc:493] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.050172 1627 raft_consensus.cc:515] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:09.050901 1627 leader_election.cc:290] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:09.052201 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_064ffeda7eaa4da6854b9395b0c98345"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.052728 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064ffeda7eaa4da6854b9395b0c98345 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.045832 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:09.056723 1600 tablet.cc:1722] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:09.061260 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5d9918fbcc644b93bee6bc87ed741463" } new_table_name: "default.Table_abf2ca58419041dbb83bfa81e6da8f08" modify_external_catalogs: true
I20251029 08:52:09.061591 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 39dc5ebeb36043e0aa5955bacd06c2ef from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:09.062150 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 74fe974a368a4afe8a5da6dfd1036400 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:09.062724 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_796d0f3f111e416aa8c95689e9d80a73" } modify_external_catalogs: true
I20251029 08:52:09.063009 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_796d0f3f111e416aa8c95689e9d80a73" } modify_external_catalogs: true
I20251029 08:52:09.065785 1176 ts_tablet_manager.cc:1950] T 64f73597d1764540bcaf472d3e890ee8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.067198 1641 tablet.cc:1722] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:09.071897 1461 raft_consensus.cc:493] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.072263 1176 tablet_replica.cc:333] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.072566 1176 raft_consensus.cc:2243] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.072960 1176 raft_consensus.cc:2272] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.073746 1176 ts_tablet_manager.cc:1916] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.075654 1178 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:09.076553 1461 raft_consensus.cc:515] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:09.077466 1461 leader_election.cc:290] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:09.078088 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28face87091b49d2971217618c95d2f5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:09.080169 720 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.080502 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:09.081698 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 302) took 1559 ms. Trace:
I20251029 08:52:09.082883 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_43572296e299444e8456983d5b1982f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.083572 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43572296e299444e8456983d5b1982f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.084637 1179 ts_tablet_manager.cc:1929] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.085361 1179 log.cc:1199] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4e808bff7558472099587a0f95c002ff
I20251029 08:52:09.086024 1179 ts_tablet_manager.cc:1950] T 4e808bff7558472099587a0f95c002ff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:09.086484 1175 rpcz_store.cc:276] 1029 08:52:07.453091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.453638 (+ 547us) service_pool.cc:225] Handling call
1029 08:52:08.978723 (+1525085us) ts_tablet_manager.cc:1151] Deleting tablet bb6b86f39c994ea5930a6a970830093f
1029 08:52:08.978733 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:08.995388 (+ 16655us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:08.998732 (+ 3344us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:09.012341 (+ 13609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524909}
I20251029 08:52:09.086941 1175 tablet_replica.cc:333] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:09.087170 1175 raft_consensus.cc:2243] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.087374 1175 raft_consensus.cc:2272] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.087393 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bb6b86f39c994ea5930a6a970830093f (table default.Table_73193142c4d04b1f83e12f1ac807a71f [id=2f82075a67804ac48d6817dd2924b9a8]) successfully deleted
I20251029 08:52:09.088191 1175 ts_tablet_manager.cc:1916] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.089874 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4e808bff7558472099587a0f95c002ff (table default.Table_ba1332d26ee0440386a364dcbfabd776 [id=f4e5b94554134c6e863ccd7bff92a249]) successfully deleted
I20251029 08:52:09.090992 1183 tablet_bootstrap.cc:492] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:09.091253 1183 ts_tablet_manager.cc:1403] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.003s
I20251029 08:52:09.091809 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4cdf03acb92449d847aec897868b967" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:09.094090 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0ff0658e0274cf49c389ad602d91b03 has no consensus state
I20251029 08:52:09.094123 1178 tablet_bootstrap.cc:654] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.094589 1183 raft_consensus.cc:359] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:09.122309 1183 raft_consensus.cc:385] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.123023 1183 raft_consensus.cc:740] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.123579 1183 consensus_queue.cc:260] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:09.133988 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24834985a71349b9865840e9b5537d72 has no consensus state
I20251029 08:52:09.134919 1183 ts_tablet_manager.cc:1434] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251029 08:52:09.143632 1605 tablet.cc:1722] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:09.148240 1183 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:09.148978 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]). Expected version 1 got 0
I20251029 08:52:09.149284 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]). Expected version 2 got 0
I20251029 08:52:09.149495 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]). Expected version 2 got 0
I20251029 08:52:09.149523 1641 tablet.cc:1717] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:09.150538 1179 tablet_replica.cc:333] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:09.150744 1179 raft_consensus.cc:2243] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.152487 1179 raft_consensus.cc:2272] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.153623 594 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543] alter complete (version 2)
I20251029 08:52:09.154259 1183 tablet_bootstrap.cc:654] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.155432 1179 ts_tablet_manager.cc:1916] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.159303 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.161115 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.161571 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 39fc9612689b46d498d6092bd552da4e (DEFAULT_TABLE table=default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.161691 1641 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:09.162066 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.162153 1051 tablet_service.cc:1505] Processing CreateTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 (DEFAULT_TABLE table=default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.162624 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 713e102d9e1d40148636443a2260ccbc (DEFAULT_TABLE table=default.Table_b1c17c659dfe49f8835fb51a638b7b53 [id=f1436eb4acb3417e8a547ef8f0203d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.163059 1043 tablet_service.cc:1505] Processing CreateTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 (DEFAULT_TABLE table=default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.163496 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 85b300f17a704f7dba1aa3724fcc2de7 (DEFAULT_TABLE table=default.Table_f159f31991334c3cac52a8b4818345e8 [id=8b155393505643e9b1b66a0a8ac5e4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.161217 1042 tablet_service.cc:1505] Processing CreateTablet for tablet b0ff0658e0274cf49c389ad602d91b03 (DEFAULT_TABLE table=default.Table_3d67d4e4f61646fe8e8864734badbc5d [id=3daf859c83c740d7a701a8bbd8aa2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.164309 1176 ts_tablet_manager.cc:1929] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.164448 1176 log.cc:1199] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:09.164840 1176 ts_tablet_manager.cc:1950] T 4deeec492a2545ca86583c6ea899a0ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.165331 1176 tablet_replica.cc:333] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.165491 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.165591 1176 raft_consensus.cc:2243] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.165838 1176 raft_consensus.cc:2272] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.166162 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.166406 1176 ts_tablet_manager.cc:1916] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.166888 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.167404 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.167966 750 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.168373 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:09.170401 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:09.170616 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_cbc899e7cb9244738c40f175d924ad40 [id=63588e39919a481c856255fd848bbf98]). Expected version 2 got 0
I20251029 08:52:09.170825 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1e96e0acdb10410a89f8f34e1f44deff" } modify_external_catalogs: true
I20251029 08:52:09.171146 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_42cce27285944566a9037e99c90de248" } modify_external_catalogs: true
I20251029 08:52:09.171655 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 215cd666c71a4179a19e298868a88e96
I20251029 08:52:09.172142 592 catalog_manager.cc:6719] ReplaceTablet: tablet 215cd666c71a4179a19e298868a88e96 deleted and replaced by tablet af913080bfe44a86a76704ffa32cd9e1
I20251029 08:52:09.172382 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_42cce27285944566a9037e99c90de248" } modify_external_catalogs: true
I20251029 08:52:09.172453 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.172554 750 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.173143 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.173231 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.173674 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1e96e0acdb10410a89f8f34e1f44deff" } modify_external_catalogs: true
I20251029 08:52:09.174486 750 tablet_service.cc:1505] Processing CreateTablet for tablet b0ff0658e0274cf49c389ad602d91b03 (DEFAULT_TABLE table=default.Table_3d67d4e4f61646fe8e8864734badbc5d [id=3daf859c83c740d7a701a8bbd8aa2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.174901 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ff0658e0274cf49c389ad602d91b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.177412 1176 ts_tablet_manager.cc:1929] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.177595 1176 log.cc:1199] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:09.177619 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c630c3859f114c0d871851d8f87a6544" } modify_external_catalogs: true
I20251029 08:52:09.177793 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c630c3859f114c0d871851d8f87a6544" } modify_external_catalogs: true
I20251029 08:52:09.178025 1176 ts_tablet_manager.cc:1950] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.178174 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13d81683d3304abba1837078e6685abd" } modify_external_catalogs: true
I20251029 08:52:09.178308 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13d81683d3304abba1837078e6685abd" } modify_external_catalogs: true
I20251029 08:52:09.178895 1176 tablet_replica.cc:333] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.179102 1176 raft_consensus.cc:2243] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.181980 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652903cbbdb042d780e7d71fc2c0ab4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.182236 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ff0658e0274cf49c389ad602d91b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.182482 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa411391c63146e7acb8b4c6edbe12a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.182704 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b300f17a704f7dba1aa3724fcc2de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.186769 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd0c43367a8c47f390cc61d2764d61e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.188299 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713e102d9e1d40148636443a2260ccbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.189069 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fc9612689b46d498d6092bd552da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.194495 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bd0c43367a8c47f390cc61d2764d61e4
I20251029 08:52:09.194603 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a79a91d787244124b45b7989b3c08c08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.195223 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a79a91d787244124b45b7989b3c08c08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.195341 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c8cb0c6c6a1c4877a89476d4822f48db
I20251029 08:52:09.198037 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.198712 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 74fe974a368a4afe8a5da6dfd1036400
I20251029 08:52:09.199256 590 catalog_manager.cc:6719] ReplaceTablet: tablet 74fe974a368a4afe8a5da6dfd1036400 deleted and replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9
I20251029 08:52:09.199786 1177 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:09.199976 1177 ts_tablet_manager.cc:1403] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.254s user 0.016s sys 0.029s
I20251029 08:52:09.199951 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 930314c8f90f462990f93fab8f7790c2 (DEFAULT_TABLE table=default.Table_74a8940fc31c4282944999b26eb6ce83 [id=5691028224ad4d21b0c0ea667274b8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.200233 923 raft_consensus.cc:2468] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:09.200389 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930314c8f90f462990f93fab8f7790c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.198731 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.205713 1178 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:09.205904 1178 ts_tablet_manager.cc:1403] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.188s user 0.008s sys 0.000s
I20251029 08:52:09.208786 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ea13e9ea2c11479ea86e3fd7f2a9e59c
I20251029 08:52:09.209842 594 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890] alter complete (version 1)
I20251029 08:52:09.216388 754 tablet_service.cc:1505] Processing CreateTablet for tablet 39fc9612689b46d498d6092bd552da4e (DEFAULT_TABLE table=default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.216908 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fc9612689b46d498d6092bd552da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.221817 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_cbc899e7cb9244738c40f175d924ad40 [id=63588e39919a481c856255fd848bbf98]). Expected version 2 got 1
I20251029 08:52:09.222075 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]). Expected version 2 got 0
I20251029 08:52:09.222496 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4b55ae84c8d441459a5931abda471f6c
I20251029 08:52:09.226881 754 tablet_service.cc:1505] Processing CreateTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 (DEFAULT_TABLE table=default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.228325 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa411391c63146e7acb8b4c6edbe12a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.228925 901 tablet_service.cc:1505] Processing CreateTablet for tablet b0ff0658e0274cf49c389ad602d91b03 (DEFAULT_TABLE table=default.Table_3d67d4e4f61646fe8e8864734badbc5d [id=3daf859c83c740d7a701a8bbd8aa2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.229182 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:09.231076 889 tablet_service.cc:1505] Processing CreateTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 (DEFAULT_TABLE table=default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.232628 888 tablet_service.cc:1505] Processing CreateTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 (DEFAULT_TABLE table=default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.233165 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.233635 897 tablet_service.cc:1505] Processing CreateTablet for tablet 85b300f17a704f7dba1aa3724fcc2de7 (DEFAULT_TABLE table=default.Table_f159f31991334c3cac52a8b4818345e8 [id=8b155393505643e9b1b66a0a8ac5e4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.236701 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b55ae84c8d441459a5931abda471f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.236861 902 tablet_service.cc:1505] Processing CreateTablet for tablet 39fc9612689b46d498d6092bd552da4e (DEFAULT_TABLE table=default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.236539 1655 raft_consensus.cc:493] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.237488 891 tablet_service.cc:1505] Processing CreateTablet for tablet 713e102d9e1d40148636443a2260ccbc (DEFAULT_TABLE table=default.Table_b1c17c659dfe49f8835fb51a638b7b53 [id=f1436eb4acb3417e8a547ef8f0203d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.237524 1655 raft_consensus.cc:515] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:09.237725 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ff0658e0274cf49c389ad602d91b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.238746 1655 leader_election.cc:290] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:09.238843 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.238140 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.239140 890 tablet_service.cc:1552] Processing DeleteTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.239252 905 tablet_service.cc:1505] Processing CreateTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (DEFAULT_TABLE table=default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.240163 896 tablet_service.cc:1505] Processing CreateTablet for tablet 930314c8f90f462990f93fab8f7790c2 (DEFAULT_TABLE table=default.Table_74a8940fc31c4282944999b26eb6ce83 [id=5691028224ad4d21b0c0ea667274b8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.240152 1178 raft_consensus.cc:359] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.239565 894 tablet_service.cc:1505] Processing CreateTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.240483 1178 raft_consensus.cc:385] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.240578 1178 raft_consensus.cc:740] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.240796 1178 consensus_queue.cc:260] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.241400 1178 ts_tablet_manager.cc:1434] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20251029 08:52:09.241690 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.242506 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.249468 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.249719 996 leader_election.cc:304] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:09.250425 1177 raft_consensus.cc:359] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.250721 1177 raft_consensus.cc:385] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.250828 1177 raft_consensus.cc:740] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.251076 1177 consensus_queue.cc:260] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.251678 1177 ts_tablet_manager.cc:1434] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251029 08:52:09.253352 759 tablet_service.cc:1505] Processing CreateTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.253432 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6fe27de4b7664cd5907bfb6507779a9b" } new_table_name: "default.Table_508e306ea26d403299fafa0a0cd2419c" modify_external_catalogs: true
I20251029 08:52:09.253504 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_78b0f14f774845d5bd92e0902a2c79fa" } new_table_name: "default.Table_abb3201c55f04e0083c3515dd7f64bee" modify_external_catalogs: true
I20251029 08:52:09.253835 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652903cbbdb042d780e7d71fc2c0ab4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.255148 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_538d255b7c7e4f9fbe5f5ee8236746b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.255954 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_17a9174cc94c45f1b499e06db7630b6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.256486 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:09.256541 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17a9174cc94c45f1b499e06db7630b6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.251999 1177 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:09.252288 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.252337 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa411391c63146e7acb8b4c6edbe12a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.252758 890 tablet_service.cc:1505] Processing CreateTablet for tablet 43c3074caee74069a8a7f1b87721d726 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.258168 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652903cbbdb042d780e7d71fc2c0ab4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.258538 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c3074caee74069a8a7f1b87721d726. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.258749 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930314c8f90f462990f93fab8f7790c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.261787 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd0c43367a8c47f390cc61d2764d61e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.252770 1605 tablet.cc:1717] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:09.265456 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8f9777707145473e941f784ae1963744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.266148 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9777707145473e941f784ae1963744 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.266752 1627 raft_consensus.cc:2804] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:09.266813 1071 raft_consensus.cc:3060] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:09.266906 1627 raft_consensus.cc:493] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:09.267863 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_57fcb0952cf24438aadd4476f7f120a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.268545 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57fcb0952cf24438aadd4476f7f120a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.269315 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:40435: OK, instance_id { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" instance_seqno: 1761727927631550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 08:52:09.269557 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 08:52:09.270305 1178 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:09.273063 1178 tablet_bootstrap.cc:654] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.273744 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:09.273850 1619 log.cc:927] Time spent T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251029 08:52:09.273944 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:09.275197 1627 raft_consensus.cc:3060] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:09.275894 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b55ae84c8d441459a5931abda471f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.278257 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (DEFAULT_TABLE table=default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.278728 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.273304 1176 raft_consensus.cc:2272] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.273504 750 tablet_service.cc:1505] Processing CreateTablet for tablet 713e102d9e1d40148636443a2260ccbc (DEFAULT_TABLE table=default.Table_b1c17c659dfe49f8835fb51a638b7b53 [id=f1436eb4acb3417e8a547ef8f0203d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.277662 759 tablet_service.cc:1505] Processing CreateTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 (DEFAULT_TABLE table=default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.279464 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd0c43367a8c47f390cc61d2764d61e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.279644 1176 ts_tablet_manager.cc:1916] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:09.280130 720 leader_election.cc:343] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.255551 893 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.267614 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fc9612689b46d498d6092bd552da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.255475 1175 ts_tablet_manager.cc:1929] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:09.283381 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 026eef7a3cba40d0a2c090485cea55e0 consensus state has no leader
I20251029 08:52:09.285441 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 43c3074caee74069a8a7f1b87721d726 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.285220 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.287659 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5c54cf056b2f4d76865c7eba3b88f6f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.288300 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.289685 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.292187 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.292898 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 026eef7a3cba40d0a2c090485cea55e0 (DEFAULT_TABLE table=default.Table_78b0f14f774845d5bd92e0902a2c79fa [id=559cd9144082414dade2ac5d73a7057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.292534 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713e102d9e1d40148636443a2260ccbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.295771 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.296062 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3222378209e141649a30f5c940ae2c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.296586 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c3074caee74069a8a7f1b87721d726. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.296689 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3222378209e141649a30f5c940ae2c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.297561 748 tablet_service.cc:1505] Processing CreateTablet for tablet 85b300f17a704f7dba1aa3724fcc2de7 (DEFAULT_TABLE table=default.Table_f159f31991334c3cac52a8b4818345e8 [id=8b155393505643e9b1b66a0a8ac5e4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.297996 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cbc899e7cb9244738c40f175d924ad40" } new_table_name: "default.Table_0f39ad7a803c4432ada5c9cf85ae0632" modify_external_catalogs: true
I20251029 08:52:09.298274 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_74a8940fc31c4282944999b26eb6ce83" } new_table_name: "default.Table_11e47ff2a71e439ebfb0e9ba79a95267" modify_external_catalogs: true
I20251029 08:52:09.298403 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026eef7a3cba40d0a2c090485cea55e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.299309 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.302187 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b300f17a704f7dba1aa3724fcc2de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.316185 756 tablet_service.cc:1552] Processing DeleteTablet for tablet c8cb0c6c6a1c4877a89476d4822f48db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.254918 1661 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:09.325879 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:09.329957 1627 raft_consensus.cc:515] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:09.334679 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b300f17a704f7dba1aa3724fcc2de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.335312 1353 raft_consensus.cc:493] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.336154 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3521ed8aec364fee9b3ee28e76464a11" } new_table_name: "default.Table_6f92afb6d07b43a3a12cc959d139105a" modify_external_catalogs: true
I20251029 08:52:09.337745 1353 raft_consensus.cc:515] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.337821 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a (DEFAULT_TABLE table=default.Table_b60d8ff960f148d586e443888f754174 [id=3b0c410e5d1e4d33ab2f50cff45fb590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.340168 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713e102d9e1d40148636443a2260ccbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.341341 1043 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4d793f10147fba5e4f44474575bf0 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.342518 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4cdf03acb92449d847aec897868b967" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:09.342736 923 raft_consensus.cc:3060] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:09.344300 1175 log.cc:1199] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a61235da8c7a4f31b29d5d50c53c347d
I20251029 08:52:09.345485 1175 ts_tablet_manager.cc:1950] T a61235da8c7a4f31b29d5d50c53c347d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:09.345932 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 304) took 1892 ms. Trace:
I20251029 08:52:09.346038 1175 rpcz_store.cc:276] 1029 08:52:07.453284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.454086 (+ 802us) service_pool.cc:225] Handling call
1029 08:52:09.086860 (+1632774us) ts_tablet_manager.cc:1151] Deleting tablet a61235da8c7a4f31b29d5d50c53c347d
1029 08:52:09.086873 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.203820 (+116947us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:09.255340 (+ 51520us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:09.345869 (+ 90529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632579}
I20251029 08:52:09.346285 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bea157c6e385431ab7527954f34028d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.346550 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 310) took 1846 ms. Trace:
I20251029 08:52:09.346673 1175 rpcz_store.cc:276] 1029 08:52:07.500104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.504098 (+ 3994us) service_pool.cc:225] Handling call
1029 08:52:09.346345 (+1842247us) ts_tablet_manager.cc:1151] Deleting tablet 5a7a673ba4fd4abcbfc3493c38dcc257
1029 08:52:09.346356 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.346489 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1842002}
W20251029 08:52:09.346926 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bea157c6e385431ab7527954f34028d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.347074 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 311) took 1756 ms. Trace:
I20251029 08:52:09.347167 1175 rpcz_store.cc:276] 1029 08:52:07.590254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.592318 (+ 2064us) service_pool.cc:225] Handling call
1029 08:52:09.346910 (+1754592us) ts_tablet_manager.cc:1151] Deleting tablet fd25eeb89e574297a92d8e978a79d768
1029 08:52:09.346918 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.347010 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732263}
I20251029 08:52:09.347576 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 313) took 1732 ms. Trace:
I20251029 08:52:09.347920 1175 rpcz_store.cc:276] 1029 08:52:07.614988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.615064 (+ 76us) service_pool.cc:225] Handling call
1029 08:52:09.347372 (+1732308us) ts_tablet_manager.cc:1151] Deleting tablet bc9c221eaa9a4721ba25761206a13645
1029 08:52:09.347379 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.347512 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732041}
I20251029 08:52:09.348323 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 312) took 1733 ms. Trace:
I20251029 08:52:09.348417 1175 rpcz_store.cc:276] 1029 08:52:07.614821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.615414 (+ 593us) service_pool.cc:225] Handling call
1029 08:52:09.348165 (+1732751us) ts_tablet_manager.cc:1151] Deleting tablet c572c68b32d443b2a019e1f4bf2d82fe
1029 08:52:09.348175 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.348267 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728863}
I20251029 08:52:09.348773 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 314) took 1733 ms. Trace:
I20251029 08:52:09.348866 1175 rpcz_store.cc:276] 1029 08:52:07.615177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.615376 (+ 199us) service_pool.cc:225] Handling call
1029 08:52:09.348625 (+1733249us) ts_tablet_manager.cc:1151] Deleting tablet f33c0bbf104046ccb86ff313f08da22a
1029 08:52:09.348636 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.348715 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725431}
I20251029 08:52:09.349143 1175 tablet_replica.cc:333] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:09.349762 893 tablet_service.cc:1505] Processing CreateTablet for tablet 026eef7a3cba40d0a2c090485cea55e0 (DEFAULT_TABLE table=default.Table_78b0f14f774845d5bd92e0902a2c79fa [id=559cd9144082414dade2ac5d73a7057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.350047 895 tablet_service.cc:1505] Processing CreateTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 (DEFAULT_TABLE table=default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.350361 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667c71a8dfd54c05ab56e845653c0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.351053 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4d793f10147fba5e4f44474575bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.359134 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4d57a59684723bfd27c7dda302ef3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.361145 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 930314c8f90f462990f93fab8f7790c2 consensus state has no leader
I20251029 08:52:09.363893 1183 tablet_bootstrap.cc:492] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:09.364765 1627 leader_election.cc:290] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:09.364753 900 tablet_service.cc:1505] Processing CreateTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a (DEFAULT_TABLE table=default.Table_b60d8ff960f148d586e443888f754174 [id=3b0c410e5d1e4d33ab2f50cff45fb590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.365067 1183 ts_tablet_manager.cc:1403] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.216s user 0.008s sys 0.006s
I20251029 08:52:09.367121 1183 raft_consensus.cc:359] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.367569 1183 raft_consensus.cc:385] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.367681 1183 raft_consensus.cc:740] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.368125 1183 consensus_queue.cc:260] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.368662 1183 ts_tablet_manager.cc:1434] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251029 08:52:09.369009 1183 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:09.370791 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_433d1398ff1f4ba88a75224fe2e4d069" } modify_external_catalogs: true
I20251029 08:52:09.371003 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_433d1398ff1f4ba88a75224fe2e4d069" } modify_external_catalogs: true
W20251029 08:52:09.373987 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2db77250e3ef489e8ab7dbcfcbb54385 consensus state has no leader
I20251029 08:52:09.375070 1049 tablet_service.cc:1505] Processing CreateTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 (DEFAULT_TABLE table=default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.375759 1179 ts_tablet_manager.cc:1929] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.375829 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4d57a59684723bfd27c7dda302ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.375900 1179 log.cc:1199] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b273c488f33c498890f1739380b18239
I20251029 08:52:09.377507 892 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4d793f10147fba5e4f44474575bf0 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.378997 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026eef7a3cba40d0a2c090485cea55e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.379767 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4d793f10147fba5e4f44474575bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.380489 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667c71a8dfd54c05ab56e845653c0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.383195 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:09.383281 1179 ts_tablet_manager.cc:1950] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:09.383652 1353 leader_election.cc:290] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:09.384316 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71ba5e1248dc4d4a8a4d7bb05b661a3f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:09.384828 861 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.386207 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (DEFAULT_TABLE table=default.Table_bd00dc7aa1464015aafad884320e9740 [id=b328433950f0429bb15733b81ea887df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.386631 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.393950 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.394392 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.396350 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 7be4bcf9e3324bd884e946fae20648fd (DEFAULT_TABLE table=default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.396767 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be4bcf9e3324bd884e946fae20648fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.400347 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (DEFAULT_TABLE table=default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.400569 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a483eeed2297411daf8f8908b370db56" } modify_external_catalogs: true
I20251029 08:52:09.400758 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.400820 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a483eeed2297411daf8f8908b370db56" } modify_external_catalogs: true
I20251029 08:52:09.401397 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_58fa7040e0e2467588dd8b5a0a2eba4b" } modify_external_catalogs: true
I20251029 08:52:09.401576 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_58fa7040e0e2467588dd8b5a0a2eba4b" } modify_external_catalogs: true
I20251029 08:52:09.393951 1039 tablet_service.cc:1505] Processing CreateTablet for tablet ba201a54466240129d82a096f9316242 (DEFAULT_TABLE table=default.Table_212e08a1f0ed463f9333cad96a91e4bd [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.404721 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba201a54466240129d82a096f9316242. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.408540 1042 tablet_service.cc:1505] Processing CreateTablet for tablet aa4f97075b3941aaa7cda3f9413e867f (DEFAULT_TABLE table=default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.409430 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4f97075b3941aaa7cda3f9413e867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.418334 1433 raft_consensus.cc:493] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.418546 1433 raft_consensus.cc:515] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.418810 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.419014 923 raft_consensus.cc:2468] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:09.419232 1433 leader_election.cc:290] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:09.419405 903 tablet_service.cc:1505] Processing CreateTablet for tablet aa4f97075b3941aaa7cda3f9413e867f (DEFAULT_TABLE table=default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.420308 899 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (DEFAULT_TABLE table=default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.420213 887 tablet_service.cc:1505] Processing CreateTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.420931 904 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.421030 1175 raft_consensus.cc:2243] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.421262 1070 raft_consensus.cc:3060] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:09.421298 904 tablet_service.cc:1505] Processing CreateTablet for tablet ba201a54466240129d82a096f9316242 (DEFAULT_TABLE table=default.Table_212e08a1f0ed463f9333cad96a91e4bd [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.421443 1175 raft_consensus.cc:2272] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.421707 898 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (DEFAULT_TABLE table=default.Table_bd00dc7aa1464015aafad884320e9740 [id=b328433950f0429bb15733b81ea887df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.422012 1175 ts_tablet_manager.cc:1916] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.422204 886 tablet_service.cc:1505] Processing CreateTablet for tablet 7be4bcf9e3324bd884e946fae20648fd (DEFAULT_TABLE table=default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.422394 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cf4be2bf29e44a4bbb908e06223cc48d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.422991 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf4be2bf29e44a4bbb908e06223cc48d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.423812 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_91e03c5d65a24eda8ba54f91a6be6081" } modify_external_catalogs: true
I20251029 08:52:09.424285 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_80786b11f8074ce283d692ae22d9e92f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.425012 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cf906c73dc84403bcc3dece74a822bd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:09.425761 861 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:09.427292 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80786b11f8074ce283d692ae22d9e92f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.428541 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a61235da8c7a4f31b29d5d50c53c347d (table default.Table_07a54a54b70a4f0aaea54ff93aa4bb65 [id=33b35dcacd094751bc02f13b4a677224]) successfully deleted
W20251029 08:52:09.429494 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a7a673ba4fd4abcbfc3493c38dcc257 already in progress: creating tablet
W20251029 08:52:09.429744 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fd25eeb89e574297a92d8e978a79d768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd25eeb89e574297a92d8e978a79d768 already in progress: creating tablet
I20251029 08:52:09.429144 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_91e03c5d65a24eda8ba54f91a6be6081" } modify_external_catalogs: true
W20251029 08:52:09.431066 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bc9c221eaa9a4721ba25761206a13645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc9c221eaa9a4721ba25761206a13645 already in progress: creating tablet
W20251029 08:52:09.431332 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c572c68b32d443b2a019e1f4bf2d82fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c572c68b32d443b2a019e1f4bf2d82fe already in progress: creating tablet
W20251029 08:52:09.432188 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f33c0bbf104046ccb86ff313f08da22a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f33c0bbf104046ccb86ff313f08da22a already in progress: creating tablet
I20251029 08:52:09.432736 1177 tablet_bootstrap.cc:654] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.433283 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1d27485babbd4ce5afd764a5d187d095" } modify_external_catalogs: true
I20251029 08:52:09.433473 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1d27485babbd4ce5afd764a5d187d095" } modify_external_catalogs: true
I20251029 08:52:09.436319 996 leader_election.cc:304] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:09.436717 1438 raft_consensus.cc:2804] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:09.436870 1438 raft_consensus.cc:697] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:09.437117 1438 consensus_queue.cc:237] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:09.442531 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.446636 1179 tablet_replica.cc:333] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:09.447593 1179 raft_consensus.cc:2243] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.448479 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.316809 749 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.452597 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.452795 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.454207 757 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.454401 746 tablet_service.cc:1505] Processing CreateTablet for tablet 43c3074caee74069a8a7f1b87721d726 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.456773 763 tablet_service.cc:1505] Processing CreateTablet for tablet 930314c8f90f462990f93fab8f7790c2 (DEFAULT_TABLE table=default.Table_74a8940fc31c4282944999b26eb6ce83 [id=5691028224ad4d21b0c0ea667274b8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.457653 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b55ae84c8d441459a5931abda471f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.458242 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.458448 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c3074caee74069a8a7f1b87721d726. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.459129 753 tablet_service.cc:1505] Processing CreateTablet for tablet 026eef7a3cba40d0a2c090485cea55e0 (DEFAULT_TABLE table=default.Table_abb3201c55f04e0083c3515dd7f64bee [id=559cd9144082414dade2ac5d73a7057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.459738 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:09.459887 861 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4cf906c73dc84403bcc3dece74a822bd
I20251029 08:52:09.460742 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.462921 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.462996 861 leader_election.cc:304] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:09.463930 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930314c8f90f462990f93fab8f7790c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.464920 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b273c488f33c498890f1739380b18239 (table default.Table_dc9e5c781bc0463d9df56e2db965b1bb [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
I20251029 08:52:09.466651 1448 raft_consensus.cc:2749] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:09.466720 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be4bcf9e3324bd884e946fae20648fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.469656 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4f97075b3941aaa7cda3f9413e867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.471163 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba201a54466240129d82a096f9316242. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.473300 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.474510 888 tablet_service.cc:1505] Processing CreateTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 (DEFAULT_TABLE table=default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.476238 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
I20251029 08:52:09.477131 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5385fef404a4463595074f0c8ff822d7
I20251029 08:52:09.477900 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2fb380018f46465e82ac6223ecf8dd0d
I20251029 08:52:09.478094 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026eef7a3cba40d0a2c090485cea55e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.478587 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:09.482147 1619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=46 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 08:52:09.482173 1183 tablet_bootstrap.cc:654] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.482877 1619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap complete.
I20251029 08:52:09.484697 749 tablet_service.cc:1505] Processing CreateTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a (DEFAULT_TABLE table=default.Table_b60d8ff960f148d586e443888f754174 [id=3b0c410e5d1e4d33ab2f50cff45fb590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.485832 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b1c17c659dfe49f8835fb51a638b7b53" } new_table_name: "default.Table_09a6e417f01d453c95b382fa9533c0fc" modify_external_catalogs: true
I20251029 08:52:09.486356 757 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.492105 762 tablet_service.cc:1505] Processing CreateTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 (DEFAULT_TABLE table=default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.492581 1179 raft_consensus.cc:2272] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:09.493404 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 713e102d9e1d40148636443a2260ccbc consensus state has no leader
I20251029 08:52:09.494627 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_212e08a1f0ed463f9333cad96a91e4bd" } new_table_name: "default.Table_9c7a9b9d1c70476ebfe5556126bb0707" modify_external_catalogs: true
I20251029 08:52:09.496333 756 tablet_service.cc:1505] Processing CreateTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (DEFAULT_TABLE table=default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.496338 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (DEFAULT_TABLE table=default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9 [id=677af5a976e14a88a896241f81fe2bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.496863 1179 ts_tablet_manager.cc:1916] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.496819 755 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4d793f10147fba5e4f44474575bf0 (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.497166 1176 ts_tablet_manager.cc:1929] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.497293 1176 log.cc:1199] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:09.497676 1176 ts_tablet_manager.cc:1950] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.498425 1176 tablet_replica.cc:333] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.498636 1176 raft_consensus.cc:2243] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.498795 1176 raft_consensus.cc:2272] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.499346 1176 ts_tablet_manager.cc:1916] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.499490 757 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (DEFAULT_TABLE table=default.Table_bd00dc7aa1464015aafad884320e9740 [id=b328433950f0429bb15733b81ea887df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.500818 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667c71a8dfd54c05ab56e845653c0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.501286 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 (DEFAULT_TABLE table=default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.501363 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (DEFAULT_TABLE table=default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.504122 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc7808f8da41e3aa0af5f318cd05c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.504302 764 tablet_service.cc:1505] Processing CreateTablet for tablet 7be4bcf9e3324bd884e946fae20648fd (DEFAULT_TABLE table=default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.504659 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4d793f10147fba5e4f44474575bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.505965 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1c9cc46a4b97451594bc17d4eaf11972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.506618 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c9cc46a4b97451594bc17d4eaf11972 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.508095 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627bc8ff99394fa2abcca15d62e5ea10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.511585 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.511955 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba201a54466240129d82a096f9316242 consensus state has no leader
I20251029 08:52:09.514367 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.515273 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be4bcf9e3324bd884e946fae20648fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.519658 752 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (DEFAULT_TABLE table=default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.520068 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_faaa0ca23c704a6986107897d6cbc024"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.520758 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4d57a59684723bfd27c7dda302ef3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.521171 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faaa0ca23c704a6986107897d6cbc024 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.522526 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.522953 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.523288 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.524420 1600 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:09.524793 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627bc8ff99394fa2abcca15d62e5ea10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.524540 759 tablet_service.cc:1505] Processing CreateTablet for tablet aa4f97075b3941aaa7cda3f9413e867f (DEFAULT_TABLE table=default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.529138 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a2d401e9111d4eb98224fd875a334025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:09.529724 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]). Expected version 3 got 1
I20251029 08:52:09.533812 760 tablet_service.cc:1505] Processing CreateTablet for tablet ba201a54466240129d82a096f9316242 (DEFAULT_TABLE table=default.Table_212e08a1f0ed463f9333cad96a91e4bd [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.534082 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.535145 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:09.535739 998 leader_election.cc:343] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b273c488f33c498890f1739380b18239
I20251029 08:52:09.535811 758 tablet_service.cc:1505] Processing CreateTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 (DEFAULT_TABLE table=default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.535864 998 leader_election.cc:304] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:09.536664 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9c4c2744d70f4d1a9e52b9604edcb436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.537205 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c4c2744d70f4d1a9e52b9604edcb436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:09.529731 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2d401e9111d4eb98224fd875a334025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.541121 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 64f73597d1764540bcaf472d3e890ee8 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
W20251029 08:52:09.542467 998 raft_consensus.cc:2686] T b273c488f33c498890f1739380b18239 P 730d0f5b7abf43a293b7df13d01ce4ee: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:09.542821 998 leader_election.cc:343] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e4cdf03acb92449d847aec897868b967
I20251029 08:52:09.544176 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.547279 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.550725 905 tablet_service.cc:1505] Processing CreateTablet for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (DEFAULT_TABLE table=default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9 [id=677af5a976e14a88a896241f81fe2bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.556141 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_cc9cefdf2fc24efb990767018ca474a8 [id=334d077ad2634b9490979d41a64e7630]). Expected version 5 got 0
I20251029 08:52:09.558542 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.560076 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6f5c6155dbd84d46bccdc4c8679532b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.561209 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f5c6155dbd84d46bccdc4c8679532b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.560590 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 8991ea78321545be85d5c480841e4751 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:09.564498 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_faaa0ca23c704a6986107897d6cbc024" } new_table_name: "default.Table_5fefde56f4da40239f0ccb837befc3db" modify_external_catalogs: true
I20251029 08:52:09.564464 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c2a36b6e0bc74f5caaa6177fc6011231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.565085 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2a36b6e0bc74f5caaa6177fc6011231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.565286 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7e3d3e525fa34d06bb82e54fc6cc95af" } new_table_name: "default.Table_c9fa1d986ec94cc1be129f6622be22d3" modify_external_catalogs: true
I20251029 08:52:09.572183 1178 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:09.575728 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.575757 895 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.581548 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e67ddf5ceb148dfad8f7be9f26cff64" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:09.581884 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.581509 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63b08840d0fa40428c61a05b277ad750" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:09.587441 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8d4ea5c42591424eb1be9e9e3981cc89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.588169 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d4ea5c42591424eb1be9e9e3981cc89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.589855 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cc9cefdf2fc24efb990767018ca474a8" } new_table_name: "default.Table_d378e85a66bb4e40b69d49ec1cf86a51" modify_external_catalogs: true
I20251029 08:52:09.592873 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_31ae710df5b54ffdb4d5b6897ad25536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.593509 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31ae710df5b54ffdb4d5b6897ad25536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.595532 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:09.595644 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5fa181849474a559701fdb8c18d398b consensus state has no leader
I20251029 08:52:09.596010 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3a2964132be042848bb5e8b54e2274d1" } modify_external_catalogs: true
I20251029 08:52:09.596254 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3a2964132be042848bb5e8b54e2274d1" } modify_external_catalogs: true
I20251029 08:52:09.598680 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:09.599010 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eeabf8f23751447aa9af02363c40361d has no consensus state
I20251029 08:52:09.598726 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28face87091b49d2971217618c95d2f5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:09.600545 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 8daa66608cf94913afb94ef661b3c898 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:09.519150 1619 raft_consensus.cc:359] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:09.608412 1619 raft_consensus.cc:740] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f718bd0f00004f20a89d028a2fa8e6b9, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.614976 1619 consensus_queue.cc:260] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 3.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:09.616581 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]). Expected version 3 got 1
I20251029 08:52:09.620456 1178 ts_tablet_manager.cc:1403] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.302s user 0.008s sys 0.018s
I20251029 08:52:09.621366 1178 raft_consensus.cc:359] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:09.622537 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627bc8ff99394fa2abcca15d62e5ea10. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.626014 861 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:09.626202 861 leader_election.cc:304] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:09.627714 1448 raft_consensus.cc:2749] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:09.628477 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8991ea78321545be85d5c480841e4751
I20251029 08:52:09.628942 588 catalog_manager.cc:6719] ReplaceTablet: tablet 8991ea78321545be85d5c480841e4751 deleted and replaced by tablet 59ad04f57c29407b9d2420520c095a28
I20251029 08:52:09.631479 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc7808f8da41e3aa0af5f318cd05c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.632097 1675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:09.635113 1675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:09.634297 1619 sys_catalog.cc:565] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 08:52:09.633865 891 tablet_service.cc:1505] Processing CreateTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (DEFAULT_TABLE table=default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.641199 888 tablet_service.cc:1505] Processing CreateTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (DEFAULT_TABLE table=default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.643379 1178 raft_consensus.cc:385] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.643553 1178 raft_consensus.cc:740] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.645242 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba201a54466240129d82a096f9316242. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.646430 718 leader_election.cc:343] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.644444 1178 consensus_queue.cc:260] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:09.650120 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4f97075b3941aaa7cda3f9413e867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.650444 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.651101 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.652379 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dc4c12b3775849c8a71e324ebbc50098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.652896 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc4c12b3775849c8a71e324ebbc50098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.654086 1178 ts_tablet_manager.cc:1434] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20251029 08:52:09.655746 718 leader_election.cc:304] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:09.667027 718 raft_consensus.cc:2686] T 55842c18dbef4513a61dd9629f2e5a4c P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:09.667160 1177 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:09.667339 1448 consensus_queue.cc:1048] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:09.673293 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 63131.4us
I20251029 08:52:09.674698 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4deeec492a2545ca86583c6ea899a0ab (table default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e [id=58d71d681f8b4cd4967daf7d53eb5191]) successfully deleted
I20251029 08:52:09.683212 1627 raft_consensus.cc:493] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.678927 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a2841227adaf4c67aed78c15574f5188" } modify_external_catalogs: true
I20251029 08:52:09.682106 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_efc1d3e55082423da1deead7cae760f4" } modify_external_catalogs: true
I20251029 08:52:09.684506 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_efc1d3e55082423da1deead7cae760f4" } modify_external_catalogs: true
I20251029 08:52:09.684845 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (table default.Table_15261ce913ae430d8ea68e18a42df64a [id=a2015498bb844e87af722a69bcbf616a]) successfully deleted
W20251029 08:52:09.685073 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 1ae561a995c94fadb288b6395c55a1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae561a995c94fadb288b6395c55a1bf already in progress: creating tablet
W20251029 08:52:09.685304 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e4f80728a7a04583bada485707beb806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4f80728a7a04583bada485707beb806 already in progress: creating tablet
I20251029 08:52:09.685741 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 55842c18dbef4513a61dd9629f2e5a4c (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:09.684502 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a2841227adaf4c67aed78c15574f5188" } modify_external_catalogs: true
W20251029 08:52:09.685959 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7792ee223dbe43aa82990541caced045 because the tablet was not found. No further retry: Not found: Tablet not found: 7792ee223dbe43aa82990541caced045
I20251029 08:52:09.687064 1179 ts_tablet_manager.cc:1929] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.682243 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_63c241868b434e1d9af341e235b0554f" } modify_external_catalogs: true
I20251029 08:52:09.682363 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0622352c737244ae90d1a8ac1a960864" } modify_external_catalogs: true
I20251029 08:52:09.688920 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0622352c737244ae90d1a8ac1a960864" } modify_external_catalogs: true
I20251029 08:52:09.689571 1042 tablet_service.cc:1505] Processing CreateTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (DEFAULT_TABLE table=default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.682976 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_897033c1be1f463fb7ef32a818c87de3 [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 1 got 0
I20251029 08:52:09.690699 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 19fdc381a0d949b0b93cef399daa3021 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.691522 1044 tablet_service.cc:1505] Processing CreateTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 (DEFAULT_TABLE table=default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.692147 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:09.692591 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 6689cf0804f946c18ebc555a191549d4 (DEFAULT_TABLE table=default.Table_8f9777707145473e941f784ae1963744 [id=bef1fc3338f941f18e6ce2019247b13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.692845 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_63c241868b434e1d9af341e235b0554f" } modify_external_catalogs: true
I20251029 08:52:09.693396 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 09968d93203140fa9201928c87702ee0 (DEFAULT_TABLE table=default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.694013 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 4e213713757949caa3f504f70026b7b5 (DEFAULT_TABLE table=default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.695379 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b21589e83794d8e981ca8deeaf26b9f" } new_table_name: "default.Table_4028cb8d8c3e48c48ef5c82e7febdd54" modify_external_catalogs: true
I20251029 08:52:09.695480 1627 raft_consensus.cc:515] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.696334 1627 leader_election.cc:290] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:09.697963 1039 tablet_service.cc:1505] Processing CreateTablet for tablet a4c74b052ac94eae9e226285504c6b40 (DEFAULT_TABLE table=default.Table_a3fd4fbe8a2841d097f0f0d235f27e04 [id=32fdba980db34781a7f359080cb1e56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.698663 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.699640 1035 tablet_service.cc:1505] Processing CreateTablet for tablet c9bbf6401423411089c9070f3b03f1a7 (DEFAULT_TABLE table=default.Table_2c760e56e16a422eb0666aa83818633f [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:09.705009 996 leader_election.cc:343] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:09.706095 1041 tablet_service.cc:1505] Processing CreateTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b (DEFAULT_TABLE table=default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.707131 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 24834985a71349b9865840e9b5537d72 (DEFAULT_TABLE table=default.Table_abf2ca58419041dbb83bfa81e6da8f08 [id=0158571f3f9246aeafbfd8b8b1548482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.707317 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 657cdaad97e24611a997e26a6293ea5e (DEFAULT_TABLE table=default.Table_bea157c6e385431ab7527954f34028d4 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.708256 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e (DEFAULT_TABLE table=default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.713434 1179 log.cc:1199] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e0b459b366cb402db2e7e8fee2b3e855
I20251029 08:52:09.714092 1179 ts_tablet_manager.cc:1950] T e0b459b366cb402db2e7e8fee2b3e855 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:09.715023 1179 tablet_replica.cc:333] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:09.715222 1179 raft_consensus.cc:2243] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:09.715495 1179 pending_rounds.cc:70] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:09.715556 1179 pending_rounds.cc:77] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037598977216512 op_type: NO_OP noop_request { }
I20251029 08:52:09.715674 1179 raft_consensus.cc:2889] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:09.715767 1179 raft_consensus.cc:2272] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.716012 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ac1af8b78a4d94b0e3da14ebfffd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.716563 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fb69271e14fe3a8d0b29d0db0dc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.716081 1043 tablet_service.cc:1505] Processing CreateTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (DEFAULT_TABLE table=default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.717077 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0ef904c2fb4dfeadbf56ae659c4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.716128 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 2fafe9a0731b48b3a60032d94623681d (DEFAULT_TABLE table=default.Table_a79a91d787244124b45b7989b3c08c08 [id=683a0914751f4a4ca50cf1d4588ca225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.716180 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d3955ff79faf42bb82f764aae79b85d9 (DEFAULT_TABLE table=default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.716216 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 (DEFAULT_TABLE table=default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.722666 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24834985a71349b9865840e9b5537d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.722970 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3955ff79faf42bb82f764aae79b85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.724154 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254e62d88c3242a0a3ac7470a8005d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.725003 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fafe9a0731b48b3a60032d94623681d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.728240 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e213713757949caa3f504f70026b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.732146 1655 raft_consensus.cc:493] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.732336 1655 raft_consensus.cc:515] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:09.732998 1655 leader_election.cc:290] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:09.733321 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09968d93203140fa9201928c87702ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.734474 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6689cf0804f946c18ebc555a191549d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.739563 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fdc381a0d949b0b93cef399daa3021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.740182 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65b3b1c8d14743aca1465d95ace95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.744201 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50e2fdfc0e64516a41c90f5a56f4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.745716 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cdaad97e24611a997e26a6293ea5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.756183 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bbf6401423411089c9070f3b03f1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.760510 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c74b052ac94eae9e226285504c6b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.764181 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf29e030892d44e1ad4d1ca4f5aeab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.781392 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d (DEFAULT_TABLE table=default.Table_1d27485babbd4ce5afd764a5d187d095 [id=ccc56f591675423da2206c023b1f8853]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:09.783582 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 102653us
I20251029 08:52:09.790055 1438 raft_consensus.cc:493] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.792339 1655 consensus_queue.cc:1048] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251029 08:52:09.794763 892 tablet_service.cc:1505] Processing CreateTablet for tablet 19fdc381a0d949b0b93cef399daa3021 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.795116 1177 ts_tablet_manager.cc:1403] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.415s user 0.012s sys 0.016s
I20251029 08:52:09.796293 1177 raft_consensus.cc:359] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.796538 1177 raft_consensus.cc:385] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.796638 1177 raft_consensus.cc:740] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.796916 1177 consensus_queue.cc:260] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.797600 1177 ts_tablet_manager.cc:1434] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:09.803988 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0ef904c2fb4dfeadbf56ae659c4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.812256 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb380018f46465e82ac6223ecf8dd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.813225 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab56045bd814d59831a98618c759af4 (DEFAULT_TABLE table=default.Table_6d54c5b6b6f84dc896c7cc341921076e [id=771dd3dd3a224a8599ab86bc7801dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.813827 1179 ts_tablet_manager.cc:1916] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.814297 1178 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:09.816751 1177 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:09.817162 1438 raft_consensus.cc:515] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.818585 902 tablet_service.cc:1505] Processing CreateTablet for tablet 2fafe9a0731b48b3a60032d94623681d (DEFAULT_TABLE table=default.Table_a79a91d787244124b45b7989b3c08c08 [id=683a0914751f4a4ca50cf1d4588ca225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.828033 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fdc381a0d949b0b93cef399daa3021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.838220 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fafe9a0731b48b3a60032d94623681d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.846149 1178 tablet_bootstrap.cc:654] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.847177 1176 ts_tablet_manager.cc:1929] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.848955 893 tablet_service.cc:1505] Processing CreateTablet for tablet 6689cf0804f946c18ebc555a191549d4 (DEFAULT_TABLE table=default.Table_8f9777707145473e941f784ae1963744 [id=bef1fc3338f941f18e6ce2019247b13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.850029 897 tablet_service.cc:1505] Processing CreateTablet for tablet 4e213713757949caa3f504f70026b7b5 (DEFAULT_TABLE table=default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.850965 904 tablet_service.cc:1505] Processing CreateTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 (DEFAULT_TABLE table=default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.852483 895 tablet_service.cc:1505] Processing CreateTablet for tablet a4c74b052ac94eae9e226285504c6b40 (DEFAULT_TABLE table=default.Table_a3fd4fbe8a2841d097f0f0d235f27e04 [id=32fdba980db34781a7f359080cb1e56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.852828 896 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.853906 903 tablet_service.cc:1505] Processing CreateTablet for tablet 09968d93203140fa9201928c87702ee0 (DEFAULT_TABLE table=default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.858748 1176 log.cc:1199] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/63b08840d0fa40428c61a05b277ad750
I20251029 08:52:09.867079 1688 raft_consensus.cc:493] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:09.867610 900 tablet_service.cc:1505] Processing CreateTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 (DEFAULT_TABLE table=default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.868350 1688 raft_consensus.cc:515] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.871546 1688 leader_election.cc:290] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:09.875396 1176 ts_tablet_manager.cc:1950] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.879761 1176 tablet_replica.cc:333] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.880198 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:09.881218 1175 ts_tablet_manager.cc:1929] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.882637 898 tablet_service.cc:1505] Processing CreateTablet for tablet c9bbf6401423411089c9070f3b03f1a7 (DEFAULT_TABLE table=default.Table_2c760e56e16a422eb0666aa83818633f [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.880292 1176 raft_consensus.cc:2243] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.887918 1176 raft_consensus.cc:2272] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.888921 1176 ts_tablet_manager.cc:1916] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.898661 1175 log.cc:1199] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e4cdf03acb92449d847aec897868b967
I20251029 08:52:09.898909 1661 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:09.895421 886 tablet_service.cc:1505] Processing CreateTablet for tablet d3955ff79faf42bb82f764aae79b85d9 (DEFAULT_TABLE table=default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.908105 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6689cf0804f946c18ebc555a191549d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.910027 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65b3b1c8d14743aca1465d95ace95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.911844 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bbf6401423411089c9070f3b03f1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.913188 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e213713757949caa3f504f70026b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.922533 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50e2fdfc0e64516a41c90f5a56f4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.923029 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3955ff79faf42bb82f764aae79b85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.924045 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c74b052ac94eae9e226285504c6b40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.925812 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cc57cd5a8d3c42ba90f2011084e8b4bd consensus state has no leader
I20251029 08:52:09.926679 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254e62d88c3242a0a3ac7470a8005d74. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:09.929005 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 144205us
W20251029 08:52:09.929312 718 leader_election.cc:343] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.930527 718 leader_election.cc:304] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:09.931384 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09968d93203140fa9201928c87702ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.932178 1687 raft_consensus.cc:2749] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:09.932694 1175 ts_tablet_manager.cc:1950] T e4cdf03acb92449d847aec897868b967 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:09.932754 718 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.932934 718 leader_election.cc:304] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:09.933153 718 raft_consensus.cc:2686] T 63b08840d0fa40428c61a05b277ad750 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:09.934001 718 leader_election.cc:343] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.934145 718 leader_election.cc:304] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:09.934371 1178 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:09.934605 1178 ts_tablet_manager.cc:1403] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.141s user 0.007s sys 0.010s
I20251029 08:52:09.935948 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 316) took 2204 ms. Trace:
I20251029 08:52:09.936312 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ac1af8b78a4d94b0e3da14ebfffd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.937019 1175 rpcz_store.cc:276] 1029 08:52:07.729435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.729508 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:09.349059 (+1619551us) ts_tablet_manager.cc:1151] Deleting tablet e4cdf03acb92449d847aec897868b967
1029 08:52:09.349069 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.421312 (+ 72243us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x616000132154. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508d311100 00007f508e65eae3 00007f509121f287 00007f50912522d4 00007f5091232d70 00007f5091232afd 0000000000a8867c 00007f508148cd7f 00007f508149004b 00007f508148fec8
1029 08:52:09.659248 (+237936us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:09.842191 (+182943us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:09.933611 (+ 91420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164823936,"tablet-delete.queue_time_us":1567824}
I20251029 08:52:09.939908 1176 ts_tablet_manager.cc:1929] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:09.940130 1687 raft_consensus.cc:2749] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:09.940872 1178 raft_consensus.cc:359] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.941139 1178 raft_consensus.cc:385] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:09.941289 1178 raft_consensus.cc:740] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:09.941545 1178 consensus_queue.cc:260] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:09.942058 887 tablet_service.cc:1505] Processing CreateTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b (DEFAULT_TABLE table=default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.942484 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf29e030892d44e1ad4d1ca4f5aeab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.942847 1178 ts_tablet_manager.cc:1434] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:09.943324 1178 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:09.942030 718 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.944002 718 leader_election.cc:304] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:09.944070 899 tablet_service.cc:1505] Processing CreateTablet for tablet 24834985a71349b9865840e9b5537d72 (DEFAULT_TABLE table=default.Table_abf2ca58419041dbb83bfa81e6da8f08 [id=0158571f3f9246aeafbfd8b8b1548482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.944402 1687 raft_consensus.cc:2749] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:09.944855 718 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:09.945770 718 leader_election.cc:304] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:09.947957 1687 raft_consensus.cc:2749] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:09.948467 1033 tablet_service.cc:1505] Processing CreateTablet for tablet ce4a3503830c481797133028f5f123d9 (DEFAULT_TABLE table=default.Table_98254c96ab3c40b6a58d2295fedd0d16 [id=e1da82f219d140ad8ee2c8d2786bd40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.948765 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.948928 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 442c37e1ee724b6b96f679f05640baf9 (DEFAULT_TABLE table=default.Table_b9370f9fc0ee4f829cbd096420cf8c53 [id=b32652cc709846deb2fc0df05493bafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.949378 1047 tablet_service.cc:1505] Processing CreateTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 (DEFAULT_TABLE table=default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da [id=85dda4d8658f4607bafddcfba7b8678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.949826 1048 tablet_service.cc:1505] Processing CreateTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 (DEFAULT_TABLE table=default.Table_3222378209e141649a30f5c940ae2c23 [id=b189fe1c508d4520a69d483cb97d4886]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.950286 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 (DEFAULT_TABLE table=default.Table_a3dc21093f304d13933e68c784dbfe5e [id=91274c3e8fac4bdbbbffefa10cd73044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.950467 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.950937 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e0b459b366cb402db2e7e8fee2b3e855 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]) successfully deleted
I20251029 08:52:09.951418 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:09.951439 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet de78ea608c904aeea7f7145b9c80f45b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de78ea608c904aeea7f7145b9c80f45b already in progress: creating tablet
I20251029 08:52:09.952104 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:09.952116 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a3ba512bd4f04bd9aceb4cb210fe9689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ba512bd4f04bd9aceb4cb210fe9689 already in progress: creating tablet
W20251029 08:52:09.952632 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4e2773cae9814df79ee60ea8b0fe4e9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2773cae9814df79ee60ea8b0fe4e9d already in progress: creating tablet
I20251029 08:52:09.952658 749 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:09.953364 749 tablet_service.cc:1505] Processing CreateTablet for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (DEFAULT_TABLE table=default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9 [id=677af5a976e14a88a896241f81fe2bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.953922 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc7808f8da41e3aa0af5f318cd05c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.955708 894 tablet_service.cc:1505] Processing CreateTablet for tablet 657cdaad97e24611a997e26a6293ea5e (DEFAULT_TABLE table=default.Table_bea157c6e385431ab7527954f34028d4 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.959755 890 tablet_service.cc:1505] Processing CreateTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e (DEFAULT_TABLE table=default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.962949 749 tablet_service.cc:1505] Processing CreateTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (DEFAULT_TABLE table=default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.963485 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ac1af8b78a4d94b0e3da14ebfffd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.966442 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 317) took 2126 ms. Trace:
I20251029 08:52:09.966985 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e4cdf03acb92449d847aec897868b967 (table default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]) successfully deleted
I20251029 08:52:09.947635 1178 tablet_bootstrap.cc:654] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:09.968600 757 tablet_service.cc:1505] Processing CreateTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (DEFAULT_TABLE table=default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.969220 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0ef904c2fb4dfeadbf56ae659c4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.971292 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2c760e56e16a422eb0666aa83818633f" } new_table_name: "default.Table_8ec38d64e3d3455ab090035a692b37ff" modify_external_catalogs: true
I20251029 08:52:09.972692 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:09.973070 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:09.973516 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_87c02133f89043fb86878e61cfe8d96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.974030 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87c02133f89043fb86878e61cfe8d96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.975554 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_308a217b7d8f415f9c68846e9e95041d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.976090 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_308a217b7d8f415f9c68846e9e95041d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.977545 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c68d96e4128e496097a5c5a4740db7ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:09.978068 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c68d96e4128e496097a5c5a4740db7ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:09.984329 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24834985a71349b9865840e9b5537d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.984881 757 tablet_service.cc:1505] Processing CreateTablet for tablet 19fdc381a0d949b0b93cef399daa3021 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.985127 758 tablet_service.cc:1505] Processing CreateTablet for tablet 2fafe9a0731b48b3a60032d94623681d (DEFAULT_TABLE table=default.Table_a79a91d787244124b45b7989b3c08c08 [id=683a0914751f4a4ca50cf1d4588ca225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.985368 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fdc381a0d949b0b93cef399daa3021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.985551 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fafe9a0731b48b3a60032d94623681d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.987432 1176 log.cc:1199] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:09.987545 901 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d (DEFAULT_TABLE table=default.Table_1d27485babbd4ce5afd764a5d187d095 [id=ccc56f591675423da2206c023b1f8853]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.987928 1176 ts_tablet_manager.cc:1950] T 39dc5ebeb36043e0aa5955bacd06c2ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:09.988134 1175 rpcz_store.cc:276] 1029 08:52:07.826341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.826395 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:09.937899 (+2111504us) ts_tablet_manager.cc:1151] Deleting tablet d30183814f5e4f2ab9d71fe493eb801c
1029 08:52:09.937910 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:09.953043 (+ 15133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2087182}
I20251029 08:52:09.988929 1176 tablet_replica.cc:333] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:09.989141 1176 raft_consensus.cc:2243] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:09.989292 1176 raft_consensus.cc:2272] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:09.989835 1176 ts_tablet_manager.cc:1916] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:09.992702 756 tablet_service.cc:1505] Processing CreateTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 (DEFAULT_TABLE table=default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:09.993069 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50e2fdfc0e64516a41c90f5a56f4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:09.996753 1071 raft_consensus.cc:3060] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:10.006228 756 tablet_service.cc:1505] Processing CreateTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 (DEFAULT_TABLE table=default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.006384 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7cbce5d595364b4bad36d59fb532c237" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:10.007349 720 leader_election.cc:343] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:10.007422 753 tablet_service.cc:1505] Processing CreateTablet for tablet d3955ff79faf42bb82f764aae79b85d9 (DEFAULT_TABLE table=default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.007961 759 tablet_service.cc:1505] Processing CreateTablet for tablet 09968d93203140fa9201928c87702ee0 (DEFAULT_TABLE table=default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.008492 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4e213713757949caa3f504f70026b7b5 (DEFAULT_TABLE table=default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.008986 748 tablet_service.cc:1505] Processing CreateTablet for tablet c9bbf6401423411089c9070f3b03f1a7 (DEFAULT_TABLE table=default.Table_2c760e56e16a422eb0666aa83818633f [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.010126 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab56045bd814d59831a98618c759af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.010437 760 tablet_service.cc:1505] Processing CreateTablet for tablet a4c74b052ac94eae9e226285504c6b40 (DEFAULT_TABLE table=default.Table_a3fd4fbe8a2841d097f0f0d235f27e04 [id=32fdba980db34781a7f359080cb1e56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.010979 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.011515 764 tablet_service.cc:1505] Processing CreateTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b (DEFAULT_TABLE table=default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.012001 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3955ff79faf42bb82f764aae79b85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.015666 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09968d93203140fa9201928c87702ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.017915 1051 tablet_service.cc:1505] Processing CreateTablet for tablet af913080bfe44a86a76704ffa32cd9e1 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.018790 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bbf6401423411089c9070f3b03f1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.019006 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e213713757949caa3f504f70026b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.020300 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c74b052ac94eae9e226285504c6b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.019968 889 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab56045bd814d59831a98618c759af4 (DEFAULT_TABLE table=default.Table_6d54c5b6b6f84dc896c7cc341921076e [id=771dd3dd3a224a8599ab86bc7801dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.023834 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fb69271e14fe3a8d0b29d0db0dc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.024338 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65b3b1c8d14743aca1465d95ace95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.024992 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:10.025274 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 318) took 2162 ms. Trace:
I20251029 08:52:10.027087 1175 rpcz_store.cc:276] 1029 08:52:07.857668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:07.857797 (+ 129us) service_pool.cc:225] Handling call
1029 08:52:09.998602 (+2140805us) ts_tablet_manager.cc:1151] Deleting tablet 7c3e50018f414f4b88e0e5d90cb7c925
1029 08:52:09.998614 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.019942 (+ 21328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2128909}
I20251029 08:52:10.027540 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 330) took 1911 ms. Trace:
I20251029 08:52:10.028054 758 tablet_service.cc:1505] Processing CreateTablet for tablet 6689cf0804f946c18ebc555a191549d4 (DEFAULT_TABLE table=default.Table_8f9777707145473e941f784ae1963744 [id=bef1fc3338f941f18e6ce2019247b13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.028506 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf29e030892d44e1ad4d1ca4f5aeab3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.028385 1175 rpcz_store.cc:276] 1029 08:52:08.116140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.116424 (+ 284us) service_pool.cc:225] Handling call
1029 08:52:10.027399 (+1910975us) ts_tablet_manager.cc:1151] Deleting tablet 09759a50a82f4b758449da9fbbc1f72d
1029 08:52:10.027410 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.027511 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907015}
I20251029 08:52:10.028791 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 331) took 1912 ms. Trace:
I20251029 08:52:10.029222 752 tablet_service.cc:1505] Processing CreateTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e (DEFAULT_TABLE table=default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.029356 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cdaad97e24611a997e26a6293ea5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.029626 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fb69271e14fe3a8d0b29d0db0dc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.030174 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254e62d88c3242a0a3ac7470a8005d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.030350 1175 rpcz_store.cc:276] 1029 08:52:08.116572 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.116621 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:10.028661 (+1912040us) ts_tablet_manager.cc:1151] Deleting tablet 9702a7d8b8304b49a250c129ecf28303
1029 08:52:10.028670 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.028760 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848308}
I20251029 08:52:10.030719 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 332) took 1909 ms. Trace:
I20251029 08:52:10.030758 762 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d (DEFAULT_TABLE table=default.Table_1d27485babbd4ce5afd764a5d187d095 [id=ccc56f591675423da2206c023b1f8853]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.031093 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb380018f46465e82ac6223ecf8dd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.032184 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6689cf0804f946c18ebc555a191549d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.032944 746 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab56045bd814d59831a98618c759af4 (DEFAULT_TABLE table=default.Table_6d54c5b6b6f84dc896c7cc341921076e [id=771dd3dd3a224a8599ab86bc7801dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.033316 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab56045bd814d59831a98618c759af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.031986 1175 rpcz_store.cc:276] 1029 08:52:08.121152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.121757 (+ 605us) service_pool.cc:225] Handling call
1029 08:52:10.030586 (+1908829us) ts_tablet_manager.cc:1151] Deleting tablet 7792ee223dbe43aa82990541caced045
1029 08:52:10.030596 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.030688 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849916}
I20251029 08:52:10.034273 759 tablet_service.cc:1505] Processing CreateTablet for tablet 657cdaad97e24611a997e26a6293ea5e (DEFAULT_TABLE table=default.Table_bea157c6e385431ab7527954f34028d4 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.034597 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cdaad97e24611a997e26a6293ea5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.035593 763 tablet_service.cc:1505] Processing CreateTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 (DEFAULT_TABLE table=default.Table_611d65e05ca149d8910e6f16f99dce9c [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.035770 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab56045bd814d59831a98618c759af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.035900 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.036888 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9429f6abb50c4f66b2e1b565f303962d (DEFAULT_TABLE table=default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.036978 755 tablet_service.cc:1505] Processing CreateTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 (DEFAULT_TABLE table=default.Table_a3dc21093f304d13933e68c784dbfe5e [id=91274c3e8fac4bdbbbffefa10cd73044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.037217 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9429f6abb50c4f66b2e1b565f303962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.037429 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a355c68ecfb45ff8c396873d91268a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.038518 745 tablet_service.cc:1505] Processing CreateTablet for tablet ce4a3503830c481797133028f5f123d9 (DEFAULT_TABLE table=default.Table_98254c96ab3c40b6a58d2295fedd0d16 [id=e1da82f219d140ad8ee2c8d2786bd40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.038913 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4a3503830c481797133028f5f123d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.040041 761 tablet_service.cc:1505] Processing CreateTablet for tablet 442c37e1ee724b6b96f679f05640baf9 (DEFAULT_TABLE table=default.Table_b9370f9fc0ee4f829cbd096420cf8c53 [id=b32652cc709846deb2fc0df05493bafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.040355 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 442c37e1ee724b6b96f679f05640baf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.041175 753 tablet_service.cc:1505] Processing CreateTablet for tablet 24834985a71349b9865840e9b5537d72 (DEFAULT_TABLE table=default.Table_abf2ca58419041dbb83bfa81e6da8f08 [id=0158571f3f9246aeafbfd8b8b1548482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.041508 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24834985a71349b9865840e9b5537d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.041826 747 tablet_service.cc:1505] Processing CreateTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 (DEFAULT_TABLE table=default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da [id=85dda4d8658f4607bafddcfba7b8678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.042150 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b87d133f8a4bd1877e07a1a8758147. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.044512 1175 tablet_replica.cc:333] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:10.045228 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 63b08840d0fa40428c61a05b277ad750 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:10.045728 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 39dc5ebeb36043e0aa5955bacd06c2ef (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
I20251029 08:52:10.046079 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.046430 749 tablet_service.cc:1505] Processing CreateTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.046550 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb380018f46465e82ac6223ecf8dd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.046766 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.047911 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4e1b6b6359b437aa295b473fa42dce1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:10.048955 1177 tablet_bootstrap.cc:654] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:10.053365 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:10.056262 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
I20251029 08:52:10.056931 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2858a0baf1f04df3864e662fe0d23366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.057139 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:10.057205 1175 raft_consensus.cc:2243] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:10.057546 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2858a0baf1f04df3864e662fe0d23366 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.057605 1175 raft_consensus.cc:2272] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.057789 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_71c5f2a1b15b4d1e95c616783cd6b837" } new_table_name: "default.Table_a69aba63f41c4596a1017f38f6ed6532" modify_external_catalogs: true
I20251029 08:52:10.058475 1175 ts_tablet_manager.cc:1916] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.058966 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 14b665b63a7341d699d499ffc8cfd9f7
I20251029 08:52:10.059571 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c9fa1d986ec94cc1be129f6622be22d3" } new_table_name: "default.Table_718f9986763c4d48afe14af4b45c2f0d" modify_external_catalogs: true
W20251029 08:52:10.060822 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e687f593eead488db4bb445a63beeec7 consensus state has no leader
I20251029 08:52:10.061192 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2d71f7240c5047f882701ddaa6309a9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.061693 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d71f7240c5047f882701ddaa6309a9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.062826 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9bbf6401423411089c9070f3b03f1a7 consensus state has no leader
I20251029 08:52:10.063037 594 master_service.cc:954] ReplaceTablet: received request to replace tablet aea79bd4e6844b7abac623824cabbeeb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.063771 888 tablet_service.cc:1505] Processing CreateTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 (DEFAULT_TABLE table=default.Table_611d65e05ca149d8910e6f16f99dce9c [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.064188 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.064519 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_09897b5139274e6fb00dd3a14c51db4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.064987 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09897b5139274e6fb00dd3a14c51db4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.066256 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f40aafe7dc94851a67abd84d80d7d7b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.066243 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d782dc9366fe4e2bbc42fe62f589e7b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.066864 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.088256 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.092283 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 9429f6abb50c4f66b2e1b565f303962d (DEFAULT_TABLE table=default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.092981 1183 tablet_bootstrap.cc:492] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:10.092885 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:10.093195 1183 ts_tablet_manager.cc:1403] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.724s user 0.027s sys 0.042s
I20251029 08:52:10.094322 1183 raft_consensus.cc:359] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.094573 1183 raft_consensus.cc:385] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:10.095023 1183 raft_consensus.cc:740] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:10.095340 1183 consensus_queue.cc:260] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.096066 1183 ts_tablet_manager.cc:1434] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:10.096470 1183 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:10.092914 1438 leader_election.cc:290] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:10.094450 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 (DEFAULT_TABLE table=default.Table_611d65e05ca149d8910e6f16f99dce9c [id=b426ad17cef044b6aacc1c9da2714432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.098466 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4a3503830c481797133028f5f123d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.098944 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9429f6abb50c4f66b2e1b565f303962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.098304 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.104210 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b87d133f8a4bd1877e07a1a8758147. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.098954 998 leader_election.cc:343] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e4cdf03acb92449d847aec897868b967
W20251029 08:52:10.105649 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 65274us
W20251029 08:52:10.106765 998 leader_election.cc:343] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:10.106930 998 leader_election.cc:304] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:10.107563 998 leader_election.cc:304] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:10.107921 1627 raft_consensus.cc:2749] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:10.108166 1627 raft_consensus.cc:2749] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:10.108258 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.112104 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.112444 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.112680 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.113837 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 442c37e1ee724b6b96f679f05640baf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.116225 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a355c68ecfb45ff8c396873d91268a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.126103 720 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:10.126714 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]). Expected version 3 got 2
I20251029 08:52:10.134245 760 tablet_service.cc:1505] Processing CreateTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 (DEFAULT_TABLE table=default.Table_3222378209e141649a30f5c940ae2c23 [id=b189fe1c508d4520a69d483cb97d4886]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.134718 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c24c0d32f74435bd9d28f2415e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.149590 895 tablet_service.cc:1505] Processing CreateTablet for tablet 9429f6abb50c4f66b2e1b565f303962d (DEFAULT_TABLE table=default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.150255 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9429f6abb50c4f66b2e1b565f303962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.171454 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.173910 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dcb7fb1512fe4aa5bbb23e4dab7da311" } new_table_name: "default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c" modify_external_catalogs: true
W20251029 08:52:10.175356 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d30183814f5e4f2ab9d71fe493eb801c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30183814f5e4f2ab9d71fe493eb801c already in progress: creating tablet
W20251029 08:52:10.175668 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 09759a50a82f4b758449da9fbbc1f72d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09759a50a82f4b758449da9fbbc1f72d already in progress: creating tablet
I20251029 08:52:10.175638 751 tablet_service.cc:1505] Processing CreateTablet for tablet af913080bfe44a86a76704ffa32cd9e1 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.175865 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9702a7d8b8304b49a250c129ecf28303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9702a7d8b8304b49a250c129ecf28303 already in progress: creating tablet
W20251029 08:52:10.176047 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7792ee223dbe43aa82990541caced045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7792ee223dbe43aa82990541caced045 already in progress: creating tablet
I20251029 08:52:10.176587 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c24c0d32f74435bd9d28f2415e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.179970 757 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.180637 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af913080bfe44a86a76704ffa32cd9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.184175 899 tablet_service.cc:1505] Processing CreateTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 (DEFAULT_TABLE table=default.Table_a3dc21093f304d13933e68c784dbfe5e [id=91274c3e8fac4bdbbbffefa10cd73044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.201359 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af913080bfe44a86a76704ffa32cd9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.200366 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:10.203686 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7cbce5d595364b4bad36d59fb532c237" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:10.205506 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a355c68ecfb45ff8c396873d91268a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.207517 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.209237 1353 consensus_queue.cc:1048] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:10.210203 718 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:10.210393 718 leader_election.cc:304] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:10.212641 1183 tablet_bootstrap.cc:654] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:10.214315 892 tablet_service.cc:1505] Processing CreateTablet for tablet ce4a3503830c481797133028f5f123d9 (DEFAULT_TABLE table=default.Table_98254c96ab3c40b6a58d2295fedd0d16 [id=e1da82f219d140ad8ee2c8d2786bd40e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.215083 1070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.060s sys 0.091s
W20251029 08:52:10.219564 1070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s user 0.060s sys 0.091s
I20251029 08:52:10.221300 897 tablet_service.cc:1505] Processing CreateTablet for tablet 442c37e1ee724b6b96f679f05640baf9 (DEFAULT_TABLE table=default.Table_b9370f9fc0ee4f829cbd096420cf8c53 [id=b32652cc709846deb2fc0df05493bafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.221944 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 442c37e1ee724b6b96f679f05640baf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.224983 886 tablet_service.cc:1505] Processing CreateTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 (DEFAULT_TABLE table=default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da [id=85dda4d8658f4607bafddcfba7b8678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.226040 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b87d133f8a4bd1877e07a1a8758147. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.225582 898 tablet_service.cc:1505] Processing CreateTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (DEFAULT_TABLE table=default.Table_ddf59288eaf146a0a467f47f4911d920 [id=ed388d60c4864cd1bacb16c26cd7b07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.227442 900 tablet_service.cc:1505] Processing CreateTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 (DEFAULT_TABLE table=default.Table_3222378209e141649a30f5c940ae2c23 [id=b189fe1c508d4520a69d483cb97d4886]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.227737 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.230600 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]). Expected version 1 got 0
I20251029 08:52:10.231580 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_877007fc953a49b189bb45e236141e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.232283 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be706771a95e45e5a20a15614eadfe1a consensus state has no leader
I20251029 08:52:10.232494 896 tablet_service.cc:1505] Processing CreateTablet for tablet af913080bfe44a86a76704ffa32cd9e1 (DEFAULT_TABLE table=default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.232731 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_76059d1e191546928659040fec3bdf30" } modify_external_catalogs: true
I20251029 08:52:10.232899 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_76059d1e191546928659040fec3bdf30" } modify_external_catalogs: true
I20251029 08:52:10.233127 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af913080bfe44a86a76704ffa32cd9e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.233525 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5fa181849474a559701fdb8c18d398b consensus state has no leader
I20251029 08:52:10.233886 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2134798e3d764329b406d0ea336c2cab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.234402 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2134798e3d764329b406d0ea336c2cab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.236137 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_266c99fef9414223bd1a5a73dbbf9bd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.236636 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_266c99fef9414223bd1a5a73dbbf9bd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.237761 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aea79bd4e6844b7abac623824cabbeeb
I20251029 08:52:10.238191 594 catalog_manager.cc:6719] ReplaceTablet: tablet aea79bd4e6844b7abac623824cabbeeb deleted and replaced by tablet b1aaf233eba2461f9182e094a0115a4e
I20251029 08:52:10.238312 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.238474 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 57e2b19a227740a28ae601d77e62597b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.238518 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.238721 889 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.238873 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.238906 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 39dc5ebeb36043e0aa5955bacd06c2ef from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.239017 889 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.239180 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.239264 594 master_service.cc:954] ReplaceTablet: received request to replace tablet add0605edf5c4511ae4c8d9cb7f0babb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.239501 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4a3503830c481797133028f5f123d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.241569 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c24c0d32f74435bd9d28f2415e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.246598 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
W20251029 08:52:10.251489 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afa2bb1075c4418eb10f4493f5e73c1b consensus state has no leader
I20251029 08:52:10.252303 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:10.252898 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bc205fa7964840f9802b4bb2a9a4f954 [id=821160def5624241afdce0f734ef2411]). Expected version 2 got 0
I20251029 08:52:10.253079 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_897033c1be1f463fb7ef32a818c87de3 [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 1 got 0
I20251029 08:52:10.259298 1049 tablet_service.cc:1505] Processing CreateTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 (DEFAULT_TABLE table=default.Table_8d4ea5c42591424eb1be9e9e3981cc89 [id=1c0e2e0996134fe1811cb2e300a861ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.261531 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d3b4e5bde9434116b192d820a909c7b1" } modify_external_catalogs: true
I20251029 08:52:10.261759 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d3b4e5bde9434116b192d820a909c7b1" } modify_external_catalogs: true
I20251029 08:52:10.263113 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5c54cf056b2f4d76865c7eba3b88f6f6" } modify_external_catalogs: true
I20251029 08:52:10.263182 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fa089fce564af4b03aa323cec6cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.263301 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5c54cf056b2f4d76865c7eba3b88f6f6" } modify_external_catalogs: true
I20251029 08:52:10.271811 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:10.272388 588 catalog_manager.cc:6719] ReplaceTablet: tablet 2f40aafe7dc94851a67abd84d80d7d7b deleted and replaced by tablet 15dae38bcced496e90ca8eec280a8317
I20251029 08:52:10.272797 1179 ts_tablet_manager.cc:1929] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:10.274801 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2d71f7240c5047f882701ddaa6309a9f" } new_table_name: "default.Table_0492a662ebb94c54bc22d21a5b46c918" modify_external_catalogs: true
I20251029 08:52:10.278326 1179 log.cc:1199] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e4cdf03acb92449d847aec897868b967
I20251029 08:52:10.280364 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3dc21093f304d13933e68c784dbfe5e" } modify_external_catalogs: true
I20251029 08:52:10.280573 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3dc21093f304d13933e68c784dbfe5e" } modify_external_catalogs: true
I20251029 08:52:10.284823 1044 tablet_service.cc:1505] Processing CreateTablet for tablet fcaa3fa779824b38887ae001623ba67b (DEFAULT_TABLE table=default.Table_c2a36b6e0bc74f5caaa6177fc6011231 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.286906 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_877007fc953a49b189bb45e236141e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.290105 1687 raft_consensus.cc:2749] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:10.291378 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:10.210851 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.292187 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_81405043420f4711a665ddfcd6bb7d1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.292757 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81405043420f4711a665ddfcd6bb7d1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.294168 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fa8884bb8e64474d936c8825d1ef3545
I20251029 08:52:10.294989 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5ea68420364f46228f00542b348dc801"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.295514 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ea68420364f46228f00542b348dc801 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.297040 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:10.298041 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_671a7306577e49c98f6ec7c4317f2f31" } modify_external_catalogs: true
I20251029 08:52:10.298229 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_671a7306577e49c98f6ec7c4317f2f31" } modify_external_catalogs: true
I20251029 08:52:10.300266 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:10.300843 594 catalog_manager.cc:6719] ReplaceTablet: tablet add0605edf5c4511ae4c8d9cb7f0babb deleted and replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5
I20251029 08:52:10.301304 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_308a217b7d8f415f9c68846e9e95041d" } new_table_name: "default.Table_325e6358252a40b0bdd59c1a0664ed7e" modify_external_catalogs: true
I20251029 08:52:10.301817 757 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.302035 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet aea79bd4e6844b7abac623824cabbeeb with delete_type TABLET_DATA_DELETED (replaced by tablet b1aaf233eba2461f9182e094a0115a4e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.302866 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_81b1fe8277b3492cadd29290219544fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.303359 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81b1fe8277b3492cadd29290219544fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.303752 718 leader_election.cc:343] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:10.303913 718 leader_election.cc:304] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:10.304107 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcaa3fa779824b38887ae001623ba67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.306092 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b (DEFAULT_TABLE table=default.Table_cf4be2bf29e44a4bbb908e06223cc48d [id=48afa12a4fd24dc0918cd31e4da595b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.307188 1179 ts_tablet_manager.cc:1950] T e4cdf03acb92449d847aec897868b967 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:10.310226 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b (DEFAULT_TABLE table=default.Table_80786b11f8074ce283d692ae22d9e92f [id=c6901c2dab494740be7681afa3f4aa9f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.312769 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 663d6587a5e14519afbb0f94d691d1b4 has no consensus state
I20251029 08:52:10.312876 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391977048e5f4f0b8f62e4d5e3d2946b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.314067 1688 raft_consensus.cc:2749] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:10.315354 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 334) took 1317 ms. Trace:
I20251029 08:52:10.315361 757 tablet_service.cc:1552] Processing DeleteTablet for tablet aea79bd4e6844b7abac623824cabbeeb with delete_type TABLET_DATA_DELETED (replaced by tablet b1aaf233eba2461f9182e094a0115a4e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.315459 1179 rpcz_store.cc:276] 1029 08:52:08.994824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.994882 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:09.714954 (+720072us) ts_tablet_manager.cc:1151] Deleting tablet e4cdf03acb92449d847aec897868b967
1029 08:52:09.714960 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.040005 (+325045us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.272672 (+232667us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.312424 (+ 39752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":719837}
I20251029 08:52:10.315886 757 tablet_service.cc:1505] Processing CreateTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 (DEFAULT_TABLE table=default.Table_8d4ea5c42591424eb1be9e9e3981cc89 [id=1c0e2e0996134fe1811cb2e300a861ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.316661 1037 tablet_service.cc:1505] Processing CreateTablet for tablet eeabf8f23751447aa9af02363c40361d (DEFAULT_TABLE table=default.Table_5fefde56f4da40239f0ccb837befc3db [id=ea6733c09e8f48d2b64292168b11d69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.317960 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 342) took 1143 ms. Trace:
I20251029 08:52:10.318056 1179 rpcz_store.cc:276] 1029 08:52:09.172249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.172325 (+ 76us) service_pool.cc:225] Handling call
1029 08:52:10.315790 (+1143465us) ts_tablet_manager.cc:1151] Deleting tablet 2723cdd513ee4e8ca6773b38f785b09e
1029 08:52:10.315802 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.315916 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143183}
I20251029 08:52:10.319226 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 343) took 1145 ms. Trace:
I20251029 08:52:10.320916 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.321763 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b (DEFAULT_TABLE table=default.Table_9c4c2744d70f4d1a9e52b9604edcb436 [id=5762d46725ec489d92cc4cb914fe070b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.324738 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet fa8884bb8e64474d936c8825d1ef3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.326515 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e4cdf03acb92449d847aec897868b967 (table default.Table_4adb698e66ef4612946087c05b279db2 [id=ef5c6b7d262c4ceb9f53e5b2770df7bc]) successfully deleted
W20251029 08:52:10.326751 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2723cdd513ee4e8ca6773b38f785b09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2723cdd513ee4e8ca6773b38f785b09e already in progress: creating tablet
I20251029 08:52:10.327971 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fa089fce564af4b03aa323cec6cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.328856 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 66329e1e548949cc9d59097953288375 (DEFAULT_TABLE table=default.Table_6f5c6155dbd84d46bccdc4c8679532b0 [id=7478238b5a424beaa652e44adb59822f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.333552 1178 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:10.334878 1179 rpcz_store.cc:276] 1029 08:52:09.173097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.173152 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:10.318324 (+1145172us) ts_tablet_manager.cc:1151] Deleting tablet 215cd666c71a4179a19e298868a88e96
1029 08:52:10.318333 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.318519 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144962}
W20251029 08:52:10.337770 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9907d044daa145168417371c2bd66544 has no consensus state
W20251029 08:52:10.338151 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 215cd666c71a4179a19e298868a88e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215cd666c71a4179a19e298868a88e96 already in progress: creating tablet
I20251029 08:52:10.340447 1176 ts_tablet_manager.cc:1929] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:10.339110 755 tablet_service.cc:1505] Processing CreateTablet for tablet fcaa3fa779824b38887ae001623ba67b (DEFAULT_TABLE table=default.Table_c2a36b6e0bc74f5caaa6177fc6011231 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.341981 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 59ad04f57c29407b9d2420520c095a28 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.346383 1178 ts_tablet_manager.cc:1403] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.390s user 0.001s sys 0.002s
I20251029 08:52:10.346905 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd2e9a95b6c40be9e17397a313d249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.349514 748 tablet_service.cc:1505] Processing CreateTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b (DEFAULT_TABLE table=default.Table_cf4be2bf29e44a4bbb908e06223cc48d [id=48afa12a4fd24dc0918cd31e4da595b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.354357 1176 log.cc:1199] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/983a406aeed243f29ab322f94a0f5239
I20251029 08:52:10.360671 1176 ts_tablet_manager.cc:1950] T 983a406aeed243f29ab322f94a0f5239 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:10.361079 756 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd490da0b0e47fc965343c123a27b22 (DEFAULT_TABLE table=default.Table_1c9cc46a4b97451594bc17d4eaf11972 [id=0d9f935637a8420b985538e9de4e21ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.364249 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 345) took 1194 ms. Trace:
I20251029 08:52:10.364391 1176 rpcz_store.cc:276] 1029 08:52:09.166614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.166783 (+ 169us) service_pool.cc:225] Handling call
1029 08:52:09.988859 (+822076us) ts_tablet_manager.cc:1151] Deleting tablet 983a406aeed243f29ab322f94a0f5239
1029 08:52:09.988868 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.045041 (+ 56173us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.333914 (+288873us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.360999 (+ 27085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":821857}
I20251029 08:52:10.366700 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391977048e5f4f0b8f62e4d5e3d2946b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.369019 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 983a406aeed243f29ab322f94a0f5239 (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
I20251029 08:52:10.369941 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd490da0b0e47fc965343c123a27b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.370326 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b (DEFAULT_TABLE table=default.Table_80786b11f8074ce283d692ae22d9e92f [id=c6901c2dab494740be7681afa3f4aa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.370925 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7a355c68ecfb45ff8c396873d91268a8
I20251029 08:52:10.371529 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcaa3fa779824b38887ae001623ba67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.374230 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd2e9a95b6c40be9e17397a313d249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.377056 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7b5051ff776e4975b917db0393a1f945
I20251029 08:52:10.378075 1176 tablet_replica.cc:333] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:10.378278 1176 raft_consensus.cc:2243] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:10.384948 1050 tablet_service.cc:1505] Processing CreateTablet for tablet dbf9e36c47be4509a998be8e7138475b (DEFAULT_TABLE table=default.Table_a2d401e9111d4eb98224fd875a334025 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.386020 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.388648 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb with delete_type TABLET_DATA_DELETED (replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.388881 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.389008 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.396401 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 2dec33b502aa4916b89b878462de1ab1 (DEFAULT_TABLE table=default.Table_2858a0baf1f04df3864e662fe0d23366 [id=5f7ee394d987440c9ab0c7ad53bfebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.397018 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 (DEFAULT_TABLE table=default.Table_87c02133f89043fb86878e61cfe8d96d [id=463d151caf764effa17b75b7875c5e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.397468 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 345) took 1135 ms. Trace:
I20251029 08:52:10.397571 1179 rpcz_store.cc:276] 1029 08:52:09.204123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.204400 (+ 277us) service_pool.cc:225] Handling call
1029 08:52:10.337622 (+1133222us) ts_tablet_manager.cc:1151] Deleting tablet bd0c43367a8c47f390cc61d2764d61e4
1029 08:52:10.337634 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.339984 (+ 2350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095771}
I20251029 08:52:10.397809 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66329e1e548949cc9d59097953288375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.397837 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 (DEFAULT_TABLE table=default.Table_0492a662ebb94c54bc22d21a5b46c918 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.398144 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663d6587a5e14519afbb0f94d691d1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.398644 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeabf8f23751447aa9af02363c40361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.400524 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dec33b502aa4916b89b878462de1ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.402060 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b20d53d6214bdc81ca205dbf922d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.403822 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046bb1fcdd34d308aecddfdf0752911. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.404757 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ad04f57c29407b9d2420520c095a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.408145 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf9e36c47be4509a998be8e7138475b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.411540 1179 tablet_replica.cc:333] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:10.411761 1179 raft_consensus.cc:2243] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:10.411733 1035 tablet_service.cc:1505] Processing CreateTablet for tablet f5b52314139945dd9c2e6ec899bf06dc (DEFAULT_TABLE table=default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c [id=0a703b25b63d4578b7eff94008a90dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.415190 1179 raft_consensus.cc:2272] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.415848 1179 ts_tablet_manager.cc:1916] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.415932 1176 raft_consensus.cc:2272] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.417068 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b52314139945dd9c2e6ec899bf06dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.419493 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 85df43bf781c421688f163331de0eb2b (DEFAULT_TABLE table=default.Table_31ae710df5b54ffdb4d5b6897ad25536 [id=2e09778b9b214426bdcd4070c9f0abca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.419768 1178 raft_consensus.cc:359] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:10.420543 761 tablet_service.cc:1505] Processing CreateTablet for tablet eeabf8f23751447aa9af02363c40361d (DEFAULT_TABLE table=default.Table_5fefde56f4da40239f0ccb837befc3db [id=ea6733c09e8f48d2b64292168b11d69a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.420691 1699 log.cc:927] Time spent T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20251029 08:52:10.420796 1178 raft_consensus.cc:385] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:10.420879 1178 raft_consensus.cc:740] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:10.421134 1178 consensus_queue.cc:260] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:10.421669 1178 ts_tablet_manager.cc:1434] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.075s user 0.001s sys 0.001s
I20251029 08:52:10.422474 1176 ts_tablet_manager.cc:1916] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.423199 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd490da0b0e47fc965343c123a27b22 (DEFAULT_TABLE table=default.Table_1c9cc46a4b97451594bc17d4eaf11972 [id=0d9f935637a8420b985538e9de4e21ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.428511 1178 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:10.429629 1033 tablet_service.cc:1505] Processing CreateTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.430094 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeabf8f23751447aa9af02363c40361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.432888 763 tablet_service.cc:1505] Processing CreateTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b (DEFAULT_TABLE table=default.Table_9c4c2744d70f4d1a9e52b9604edcb436 [id=5762d46725ec489d92cc4cb914fe070b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.433066 753 tablet_service.cc:1505] Processing CreateTablet for tablet dbf9e36c47be4509a998be8e7138475b (DEFAULT_TABLE table=default.Table_a2d401e9111d4eb98224fd875a334025 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.437973 899 tablet_service.cc:1505] Processing CreateTablet for tablet fcaa3fa779824b38887ae001623ba67b (DEFAULT_TABLE table=default.Table_c2a36b6e0bc74f5caaa6177fc6011231 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.435806 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 9907d044daa145168417371c2bd66544 (DEFAULT_TABLE table=default.Table_325e6358252a40b0bdd59c1a0664ed7e [id=3bb1d67e45ea4d7fbc5f901e60cf7083]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.439018 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcaa3fa779824b38887ae001623ba67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.442605 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf9e36c47be4509a998be8e7138475b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.446039 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.446033 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_d378e85a66bb4e40b69d49ec1cf86a51 [id=334d077ad2634b9490979d41a64e7630]). Expected version 6 got 5
I20251029 08:52:10.449159 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85df43bf781c421688f163331de0eb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.450472 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5b250161ee2248cd8188be4cb32cab6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.451095 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b250161ee2248cd8188be4cb32cab6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.451895 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9907d044daa145168417371c2bd66544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.452589 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_31cc9d5473cd45dcb2a4fb9609ff132b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.453156 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31cc9d5473cd45dcb2a4fb9609ff132b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.454533 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bea157c6e385431ab7527954f34028d4" } new_table_name: "default.Table_cfc355ba254a4851867ed2eed4f56cd3" modify_external_catalogs: true
I20251029 08:52:10.455840 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c3267734790547088d7d0aabd176ab02" } new_table_name: "default.Table_82ec1f06f21147c991f019803452df8c" modify_external_catalogs: true
I20251029 08:52:10.457098 1045 tablet_service.cc:1505] Processing CreateTablet for tablet b1aaf233eba2461f9182e094a0115a4e (DEFAULT_TABLE table=default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.457154 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_55491b160d0e4e6185be2e0fee2e67a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.457278 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b20d53d6214bdc81ca205dbf922d7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.457648 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55491b160d0e4e6185be2e0fee2e67a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.458016 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 0c2072c16fce43278f1e32b9672187e7 (DEFAULT_TABLE table=default.Table_dc4c12b3775849c8a71e324ebbc50098 [id=468fee62b8ce4f0dad6500dcfe3c70c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.458552 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1aaf233eba2461f9182e094a0115a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.458910 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd490da0b0e47fc965343c123a27b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.460305 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.461112 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fb40280e4c7d421889ad47f81c4574b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.461648 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb40280e4c7d421889ad47f81c4574b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.462954 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_335bd76889fc4d0aa32a3ae0baf73890" } new_table_name: "default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4" modify_external_catalogs: true
I20251029 08:52:10.464640 1704 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:10.467674 754 tablet_service.cc:1505] Processing CreateTablet for tablet 66329e1e548949cc9d59097953288375 (DEFAULT_TABLE table=default.Table_6f5c6155dbd84d46bccdc4c8679532b0 [id=7478238b5a424beaa652e44adb59822f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.473199 1051 tablet_service.cc:1505] Processing CreateTablet for tablet d306be1d51674c2abc506d248e05a9c7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.475714 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2072c16fce43278f1e32b9672187e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.476154 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f2b34642704cd6bb966b6dcd494d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.477247 1183 tablet_bootstrap.cc:492] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
W20251029 08:52:10.479157 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 657cdaad97e24611a997e26a6293ea5e consensus state has no leader
W20251029 08:52:10.479568 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 840ff254244f43a18679232279e02d1f consensus state has no leader
I20251029 08:52:10.483780 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d306be1d51674c2abc506d248e05a9c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.483804 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_15cd2138e80e429db3004528a89ec2bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.484977 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_11a78e5b37424289a184f8bccbd263d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.485458 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11a78e5b37424289a184f8bccbd263d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.486068 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (DEFAULT_TABLE table=default.Table_81b1fe8277b3492cadd29290219544fc [id=7e89e19d5a384f8195c22592fe17e43d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.486833 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15cd2138e80e429db3004528a89ec2bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.487840 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb9ab76dcfe486e89bbe9391cab56f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.488945 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bd0c43367a8c47f390cc61d2764d61e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd0c43367a8c47f390cc61d2764d61e4 already in progress: creating tablet
I20251029 08:52:10.490016 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 (DEFAULT_TABLE table=default.Table_c68d96e4128e496097a5c5a4740db7ac [id=09dcc91cf1e045169ca0eedfe1e6dad3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.491012 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07906300f1d54d4e8fde5165ee14c44f consensus state has no leader
I20251029 08:52:10.495723 749 tablet_service.cc:1552] Processing DeleteTablet for tablet fa8884bb8e64474d936c8825d1ef3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.495939 749 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.500670 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4c3334f5e9ce4296866318c8b7fbe1eb" } modify_external_catalogs: true
I20251029 08:52:10.500893 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4c3334f5e9ce4296866318c8b7fbe1eb" } modify_external_catalogs: true
I20251029 08:52:10.500947 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dd22da054ccc4c1f87f65c41649bd1a5" } modify_external_catalogs: true
I20251029 08:52:10.501119 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dd22da054ccc4c1f87f65c41649bd1a5" } modify_external_catalogs: true
I20251029 08:52:10.502326 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510" } modify_external_catalogs: true
I20251029 08:52:10.502480 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510" } modify_external_catalogs: true
I20251029 08:52:10.503654 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 4200db822deb4ace936f561c230ac1fb (DEFAULT_TABLE table=default.Table_5ea68420364f46228f00542b348dc801 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.505535 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 657cdaad97e24611a997e26a6293ea5e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.506608 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2dec33b502aa4916b89b878462de1ab1 (DEFAULT_TABLE table=default.Table_2858a0baf1f04df3864e662fe0d23366 [id=5f7ee394d987440c9ab0c7ad53bfebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.508723 764 tablet_service.cc:1505] Processing CreateTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 (DEFAULT_TABLE table=default.Table_87c02133f89043fb86878e61cfe8d96d [id=463d151caf764effa17b75b7875c5e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.509058 745 tablet_service.cc:1505] Processing CreateTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.510226 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66329e1e548949cc9d59097953288375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.511265 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 471737aa35e94dcfaa2b3451cc8f2014 (DEFAULT_TABLE table=default.Table_81405043420f4711a665ddfcd6bb7d1b [id=427664ed0296404d9a435ff598f2d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.514398 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3fd4fbe8a2841d097f0f0d235f27e04" } new_table_name: "default.Table_d64ea78b6d1648eabf348d4da0cffae8" modify_external_catalogs: true
I20251029 08:52:10.515244 759 tablet_service.cc:1505] Processing CreateTablet for tablet f5b52314139945dd9c2e6ec899bf06dc (DEFAULT_TABLE table=default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c [id=0a703b25b63d4578b7eff94008a90dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.516009 758 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.516868 751 tablet_service.cc:1505] Processing CreateTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 (DEFAULT_TABLE table=default.Table_0492a662ebb94c54bc22d21a5b46c918 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.522590 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 31bed64a27164baa8177b87b54b5947b (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.522890 758 tablet_service.cc:1505] Processing CreateTablet for tablet 85df43bf781c421688f163331de0eb2b (DEFAULT_TABLE table=default.Table_31ae710df5b54ffdb4d5b6897ad25536 [id=2e09778b9b214426bdcd4070c9f0abca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.524587 749 tablet_service.cc:1552] Processing DeleteTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb with delete_type TABLET_DATA_DELETED (replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.524806 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.528188 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.529242 760 tablet_service.cc:1505] Processing CreateTablet for tablet 0c2072c16fce43278f1e32b9672187e7 (DEFAULT_TABLE table=default.Table_dc4c12b3775849c8a71e324ebbc50098 [id=468fee62b8ce4f0dad6500dcfe3c70c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.530009 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c74b052ac94eae9e226285504c6b40 consensus state has no leader
I20251029 08:52:10.531018 746 tablet_service.cc:1505] Processing CreateTablet for tablet b1aaf233eba2461f9182e094a0115a4e (DEFAULT_TABLE table=default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.536008 762 tablet_service.cc:1505] Processing CreateTablet for tablet 9907d044daa145168417371c2bd66544 (DEFAULT_TABLE table=default.Table_325e6358252a40b0bdd59c1a0664ed7e [id=3bb1d67e45ea4d7fbc5f901e60cf7083]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.536614 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 657cdaad97e24611a997e26a6293ea5e
I20251029 08:52:10.537063 588 catalog_manager.cc:6719] ReplaceTablet: tablet 657cdaad97e24611a997e26a6293ea5e deleted and replaced by tablet a506e35afe6a4c87901f27c434f3b348
I20251029 08:52:10.537731 1438 raft_consensus.cc:493] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:10.539059 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046bb1fcdd34d308aecddfdf0752911. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.540088 747 tablet_service.cc:1505] Processing CreateTablet for tablet 59ad04f57c29407b9d2420520c095a28 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.540077 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 59c90087c9e94665a9b3fac798304af7 (DEFAULT_TABLE table=default.Table_266c99fef9414223bd1a5a73dbbf9bd5 [id=5b45ec162f5e43d4a7976bb7f6426dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.540457 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ad04f57c29407b9d2420520c095a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.541417 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_344943c2bd4d4ea1bf1095ed952a2dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.541478 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c0ea863d5d2b4ba9ae6ed5103c5025b5" } new_table_name: "default.Table_64f9de5287d54b348a406a42a7c379ab" modify_external_catalogs: true
W20251029 08:52:10.541975 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_344943c2bd4d4ea1bf1095ed952a2dd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.542295 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2d05ded03f9d4415ada974d91d47ec0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.542764 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d05ded03f9d4415ada974d91d47ec0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.543043 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c407207334e644189fdbfcae1dc12ba9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.543450 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c407207334e644189fdbfcae1dc12ba9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.544384 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9907d044daa145168417371c2bd66544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.544770 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3a2fab467348099a4a1fec74eb5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.545451 1438 raft_consensus.cc:515] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.546206 1438 leader_election.cc:290] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:10.547266 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1aaf233eba2461f9182e094a0115a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.547226 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e2b19a227740a28ae601d77e62597b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:10.547463 922 raft_consensus.cc:2468] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:10.548187 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2072c16fce43278f1e32b9672187e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.548964 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b52314139945dd9c2e6ec899bf06dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.549861 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dec33b502aa4916b89b878462de1ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.549935 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5df6707d7a754de3b61f32cf68786b2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.550169 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85df43bf781c421688f163331de0eb2b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.550691 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5df6707d7a754de3b61f32cf68786b2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.554350 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 (DEFAULT_TABLE table=default.Table_81ab02d14c514f1f9c7622f3776b4552 [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.555344 892 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd490da0b0e47fc965343c123a27b22 (DEFAULT_TABLE table=default.Table_1c9cc46a4b97451594bc17d4eaf11972 [id=0d9f935637a8420b985538e9de4e21ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.555756 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd490da0b0e47fc965343c123a27b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.556474 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.559046 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200db822deb4ace936f561c230ac1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.561060 996 leader_election.cc:304] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:10.562626 1438 raft_consensus.cc:2804] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:10.562755 1438 raft_consensus.cc:493] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:10.564173 1438 raft_consensus.cc:3060] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:10.562914 1050 tablet_service.cc:1505] Processing CreateTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 (DEFAULT_TABLE table=default.Table_877007fc953a49b189bb45e236141e2d [id=d45481f7479f46d3bc3b014c7a437f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.565565 901 tablet_service.cc:1505] Processing CreateTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b (DEFAULT_TABLE table=default.Table_80786b11f8074ce283d692ae22d9e92f [id=c6901c2dab494740be7681afa3f4aa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.566927 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd2e9a95b6c40be9e17397a313d249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.569679 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:10.570907 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:10.571148 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 (DEFAULT_TABLE table=default.Table_c68d96e4128e496097a5c5a4740db7ac [id=09dcc91cf1e045169ca0eedfe1e6dad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.572077 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:10.572649 1183 ts_tablet_manager.cc:1403] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.385s user 0.006s sys 0.017s
W20251029 08:52:10.573307 998 leader_election.cc:343] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 57e2b19a227740a28ae601d77e62597b
I20251029 08:52:10.573779 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c5175dfc80a4536bb7ff8968998073a" } modify_external_catalogs: true
I20251029 08:52:10.573959 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c5175dfc80a4536bb7ff8968998073a" } modify_external_catalogs: true
I20251029 08:52:10.576660 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471737aa35e94dcfaa2b3451cc8f2014. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.579102 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5f9765183d954404b485e1a99c508874" } modify_external_catalogs: true
I20251029 08:52:10.579342 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5f9765183d954404b485e1a99c508874" } modify_external_catalogs: true
I20251029 08:52:10.586148 757 tablet_service.cc:1505] Processing CreateTablet for tablet d306be1d51674c2abc506d248e05a9c7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.596462 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3a2fab467348099a4a1fec74eb5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.597833 1438 raft_consensus.cc:515] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:10.599326 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbc4345cabbc476485b6a040a076ddaf consensus state has no leader
I20251029 08:52:10.599831 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9fe3e9a507441b8ba5953c0120289c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.601289 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f2b34642704cd6bb966b6dcd494d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.604792 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.605816 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aafa846cb2144cec9ce4b28e61d91961
I20251029 08:52:10.609961 1178 tablet_bootstrap.cc:654] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:10.610728 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (DEFAULT_TABLE table=default.Table_81b1fe8277b3492cadd29290219544fc [id=7e89e19d5a384f8195c22592fe17e43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.612491 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4200db822deb4ace936f561c230ac1fb (DEFAULT_TABLE table=default.Table_5ea68420364f46228f00542b348dc801 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.613131 1175 ts_tablet_manager.cc:1929] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:10.613448 1175 log.cc:1199] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/933817bf2058464f837ba96921c4ed63
I20251029 08:52:10.613989 1175 ts_tablet_manager.cc:1950] T 933817bf2058464f837ba96921c4ed63 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:10.614450 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 334) took 2490 ms. Trace:
I20251029 08:52:10.614612 1175 rpcz_store.cc:276] 1029 08:52:08.123501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.124470 (+ 969us) service_pool.cc:225] Handling call
1029 08:52:10.033785 (+1909315us) ts_tablet_manager.cc:1151] Deleting tablet 933817bf2058464f837ba96921c4ed63
1029 08:52:10.033795 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.171540 (+137745us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.612993 (+441453us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.614399 (+ 1406us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12371,"tablet-delete.queue_time_us":1840419}
I20251029 08:52:10.615489 1175 tablet_replica.cc:333] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:10.615757 1175 raft_consensus.cc:2243] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:10.616075 1175 raft_consensus.cc:2272] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.617667 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663d6587a5e14519afbb0f94d691d1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.620599 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb9ab76dcfe486e89bbe9391cab56f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.626513 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d306be1d51674c2abc506d248e05a9c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.622612 898 tablet_service.cc:1505] Processing CreateTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b (DEFAULT_TABLE table=default.Table_9c4c2744d70f4d1a9e52b9604edcb436 [id=5762d46725ec489d92cc4cb914fe070b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.623040 902 tablet_service.cc:1505] Processing CreateTablet for tablet 66329e1e548949cc9d59097953288375 (DEFAULT_TABLE table=default.Table_6f5c6155dbd84d46bccdc4c8679532b0 [id=7478238b5a424beaa652e44adb59822f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.623238 890 tablet_service.cc:1552] Processing DeleteTablet for tablet fa8884bb8e64474d936c8825d1ef3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.627302 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b20d53d6214bdc81ca205dbf922d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.627393 890 tablet_service.cc:1505] Processing CreateTablet for tablet 9907d044daa145168417371c2bd66544 (DEFAULT_TABLE table=default.Table_325e6358252a40b0bdd59c1a0664ed7e [id=3bb1d67e45ea4d7fbc5f901e60cf7083]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.627734 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9907d044daa145168417371c2bd66544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.623413 904 tablet_service.cc:1552] Processing DeleteTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb with delete_type TABLET_DATA_DELETED (replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.629325 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e2b19a227740a28ae601d77e62597b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:10.629547 922 raft_consensus.cc:3060] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:10.629469 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4e208427b94bd79bcb5c1080647731 (DEFAULT_TABLE table=default.Table_5b250161ee2248cd8188be4cb32cab6b [id=064c00240f9c48ceac41175a2b8306bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.623692 905 tablet_service.cc:1505] Processing CreateTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.630223 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f2b34642704cd6bb966b6dcd494d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.628755 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26ff78aa5f94ae49a197daf2be00aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.632126 1438 leader_election.cc:290] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:10.632836 904 tablet_service.cc:1505] Processing CreateTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 (DEFAULT_TABLE table=default.Table_87c02133f89043fb86878e61cfe8d96d [id=463d151caf764effa17b75b7875c5e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.633046 922 raft_consensus.cc:2468] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:10.633255 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046bb1fcdd34d308aecddfdf0752911. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.634375 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:10.635057 998 leader_election.cc:343] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 57e2b19a227740a28ae601d77e62597b
I20251029 08:52:10.635308 1707 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:10.623999 893 tablet_service.cc:1505] Processing CreateTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 (DEFAULT_TABLE table=default.Table_0492a662ebb94c54bc22d21a5b46c918 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.637131 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663d6587a5e14519afbb0f94d691d1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.637696 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.639405 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 15dae38bcced496e90ca8eec280a8317 (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.624260 891 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.640479 891 tablet_service.cc:1505] Processing CreateTablet for tablet f5b52314139945dd9c2e6ec899bf06dc (DEFAULT_TABLE table=default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c [id=0a703b25b63d4578b7eff94008a90dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.640821 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b52314139945dd9c2e6ec899bf06dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.624447 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.647367 887 tablet_service.cc:1505] Processing CreateTablet for tablet 85df43bf781c421688f163331de0eb2b (DEFAULT_TABLE table=default.Table_31ae710df5b54ffdb4d5b6897ad25536 [id=2e09778b9b214426bdcd4070c9f0abca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.647794 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85df43bf781c421688f163331de0eb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.624589 903 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.648203 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.648401 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.648731 903 tablet_service.cc:1505] Processing CreateTablet for tablet 0c2072c16fce43278f1e32b9672187e7 (DEFAULT_TABLE table=default.Table_dc4c12b3775849c8a71e324ebbc50098 [id=468fee62b8ce4f0dad6500dcfe3c70c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.649048 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2072c16fce43278f1e32b9672187e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.650739 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 39fc9612689b46d498d6092bd552da4e
I20251029 08:52:10.624838 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2dec33b502aa4916b89b878462de1ab1 (DEFAULT_TABLE table=default.Table_2858a0baf1f04df3864e662fe0d23366 [id=5f7ee394d987440c9ab0c7ad53bfebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.652345 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dec33b502aa4916b89b878462de1ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.652369 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.652654 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.656385 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66329e1e548949cc9d59097953288375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.657145 1040 tablet_service.cc:1505] Processing CreateTablet for tablet a506e35afe6a4c87901f27c434f3b348 (DEFAULT_TABLE table=default.Table_cfc355ba254a4851867ed2eed4f56cd3 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.658473 898 tablet_service.cc:1505] Processing CreateTablet for tablet b1aaf233eba2461f9182e094a0115a4e (DEFAULT_TABLE table=default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.658828 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1aaf233eba2461f9182e094a0115a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.659276 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 34729c3f97a44713a352bde297a80027 (DEFAULT_TABLE table=default.Table_15cd2138e80e429db3004528a89ec2bf [id=a8a2e644aa73491b9d22e81babf2e582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.659672 1175 ts_tablet_manager.cc:1916] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.625150 894 tablet_service.cc:1505] Processing CreateTablet for tablet 59ad04f57c29407b9d2420520c095a28 (DEFAULT_TABLE table=default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.660269 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ad04f57c29407b9d2420520c095a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.660261 1183 raft_consensus.cc:359] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.660627 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c90087c9e94665a9b3fac798304af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.661468 890 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 (DEFAULT_TABLE table=default.Table_c68d96e4128e496097a5c5a4740db7ac [id=09dcc91cf1e045169ca0eedfe1e6dad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.661749 1070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.2:35379 (request call id 77) took 1226 ms. Trace:
I20251029 08:52:10.661793 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3a2fab467348099a4a1fec74eb5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.662744 1037 tablet_service.cc:1505] Processing CreateTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 (DEFAULT_TABLE table=default.Table_11a78e5b37424289a184f8bccbd263d4 [id=c54d460a25104cc19f43a5517c203b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.664935 1183 raft_consensus.cc:385] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:10.625478 900 tablet_service.cc:1505] Processing CreateTablet for tablet dbf9e36c47be4509a998be8e7138475b (DEFAULT_TABLE table=default.Table_a2d401e9111d4eb98224fd875a334025 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.670392 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf9e36c47be4509a998be8e7138475b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.625839 897 tablet_service.cc:1505] Processing CreateTablet for tablet eeabf8f23751447aa9af02363c40361d (DEFAULT_TABLE table=default.Table_5fefde56f4da40239f0ccb837befc3db [id=ea6733c09e8f48d2b64292168b11d69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.673789 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15dae38bcced496e90ca8eec280a8317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.673978 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f8c4f6e0aec748b9945c2dfe3cc5247a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.674178 1183 raft_consensus.cc:740] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:10.674575 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c4f6e0aec748b9945c2dfe3cc5247a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.676152 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeabf8f23751447aa9af02363c40361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.681604 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5b250161ee2248cd8188be4cb32cab6b" } new_table_name: "default.Table_8df249abd861403db422e95addc8252c" modify_external_catalogs: true
I20251029 08:52:10.683198 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a2d401e9111d4eb98224fd875a334025" } new_table_name: "default.Table_b778f7a011e14d90a2f86ad390f9834f" modify_external_catalogs: true
I20251029 08:52:10.683939 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b200db354ad94efeb37cb69294716d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.685000 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b200db354ad94efeb37cb69294716d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.686825 895 tablet_service.cc:1505] Processing CreateTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b (DEFAULT_TABLE table=default.Table_cf4be2bf29e44a4bbb908e06223cc48d [id=48afa12a4fd24dc0918cd31e4da595b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.688324 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_564ecd352b5a4bca97c4f3dfecdaad0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.688828 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_564ecd352b5a4bca97c4f3dfecdaad0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.691774 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 933817bf2058464f837ba96921c4ed63 (table default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:10.695456 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6d54c5b6b6f84dc896c7cc341921076e" } new_table_name: "default.Table_29abde71edfc496a8c61702f28f25c63" modify_external_catalogs: true
I20251029 08:52:10.696889 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_20e164b8395a4054a8c704d344a20a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.697417 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20e164b8395a4054a8c704d344a20a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.703495 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391977048e5f4f0b8f62e4d5e3d2946b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.708853 905 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (DEFAULT_TABLE table=default.Table_81b1fe8277b3492cadd29290219544fc [id=7e89e19d5a384f8195c22592fe17e43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.713105 893 tablet_service.cc:1505] Processing CreateTablet for tablet 471737aa35e94dcfaa2b3451cc8f2014 (DEFAULT_TABLE table=default.Table_81405043420f4711a665ddfcd6bb7d1b [id=427664ed0296404d9a435ff598f2d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.713131 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d1c5d76d3bd24eca923ed08a72bbc56b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.713987 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_247e1528f531419d80295027ead897a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.716722 888 tablet_service.cc:1505] Processing CreateTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 (DEFAULT_TABLE table=default.Table_8d4ea5c42591424eb1be9e9e3981cc89 [id=1c0e2e0996134fe1811cb2e300a861ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.717465 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200db822deb4ace936f561c230ac1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.718941 902 tablet_service.cc:1505] Processing CreateTablet for tablet 31bed64a27164baa8177b87b54b5947b (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.722605 892 tablet_service.cc:1505] Processing CreateTablet for tablet d306be1d51674c2abc506d248e05a9c7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.726730 886 tablet_service.cc:1552] Processing DeleteTablet for tablet aea79bd4e6844b7abac623824cabbeeb with delete_type TABLET_DATA_DELETED (replaced by tablet b1aaf233eba2461f9182e094a0115a4e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.729455 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb9ab76dcfe486e89bbe9391cab56f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.731407 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471737aa35e94dcfaa2b3451cc8f2014. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.733028 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_877007fc953a49b189bb45e236141e2d" } new_table_name: "default.Table_c488d79d59554375912c487ac611bf0a" modify_external_catalogs: true
I20251029 08:52:10.737084 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 43c3074caee74069a8a7f1b87721d726 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:10.744143 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 62597189ecb047788b83788b514e9099 (DEFAULT_TABLE table=default.Table_31cc9d5473cd45dcb2a4fb9609ff132b [id=c14f386ee208443db44d014c6b613070]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.744395 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506e35afe6a4c87901f27c434f3b348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.745965 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fa089fce564af4b03aa323cec6cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.746912 904 tablet_service.cc:1505] Processing CreateTablet for tablet 4200db822deb4ace936f561c230ac1fb (DEFAULT_TABLE table=default.Table_5ea68420364f46228f00542b348dc801 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.747922 1043 tablet_service.cc:1505] Processing CreateTablet for tablet d444a5d489434696ae178c6d650bf8d5 (DEFAULT_TABLE table=default.Table_55491b160d0e4e6185be2e0fee2e67a5 [id=8431eac1a49a41c080964575e16345d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.748255 886 tablet_service.cc:1505] Processing CreateTablet for tablet 59c90087c9e94665a9b3fac798304af7 (DEFAULT_TABLE table=default.Table_266c99fef9414223bd1a5a73dbbf9bd5 [id=5b45ec162f5e43d4a7976bb7f6426dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.751344 1176 ts_tablet_manager.cc:1929] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:10.752599 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4200db822deb4ace936f561c230ac1fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.747387 1070 rpcz_store.cc:276] 1029 08:52:09.420738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.420894 (+ 156us) service_pool.cc:225] Handling call
1029 08:52:09.421115 (+ 221us) raft_consensus.cc:1471] Updating replica for 2 ops
1029 08:52:09.426205 (+ 5090us) raft_consensus.cc:1534] Early marking committed up to index 0
1029 08:52:09.426210 (+ 5us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1029 08:52:09.426606 (+ 396us) alter_schema_op.cc:128] START. Timestamp: 1761727928793000
1029 08:52:09.427717 (+ 1111us) log.cc:844] Serialized 199 byte log entry
1029 08:52:10.239332 (+811615us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x60c00033c440. stack: 00007f0aa8bae9f6 00007f0aa8bae7b0 00007f0aa7b8a3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f0abc76bd65 00007f0ab4324f90 00007f0aa8bd3380 0000000000a75106 00007f0aa8bf2a2a 00007f0aa8bee44c 00007f0aa8bf377e 00007f0ab430a1bc 00007f0ab43174eb 00007f0ab4317f20
1029 08:52:10.239400 (+ 68us) raft_consensus.cc:1645] Marking committed up to 2
1029 08:52:10.544208 (+304808us) log.cc:844] Serialized 14 byte log entry
1029 08:52:10.630421 (+ 86213us) raft_consensus.cc:1697] Filling consensus response to leader.
1029 08:52:10.645779 (+ 15358us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1029 08:52:10.645793 (+ 14us) raft_consensus.cc:1685] finished
1029 08:52:10.645796 (+ 3us) raft_consensus.cc:1691] UpdateReplicas() finished
1029 08:52:10.646913 (+ 1117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:09.426818 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 08:52:09.426926 (+ 108us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 08:52:09.426937 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
1029 08:52:09.426945 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 08:52:10.544923 (+1117978us) op_driver.cc:464] REPLICATION: finished
1029 08:52:10.630351 (+ 85428us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 08:52:10.663169 (+ 32818us) log.cc:844] Serialized 5 byte log entry
Metrics: {"prepare.queue_time_us":228748,"prepare.run_cpu_time_us":5,"prepare.run_wall_time_us":5,"raft.queue_time_us":12515,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":86,"spinlock_wait_cycles":4574976,"thread_start_us":1096494,"threads_started":2,"wal-append.queue_time_us":789124,"child_traces":[["op",{"apply.queue_time_us":85239,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":143,"prepare.run_wall_time_us":143,"replication_time_us":1118284,"spinlock_wait_cycles":86912,"thread_start_us":85132,"threads_started":1}]]}
W20251029 08:52:10.755347 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c4e208427b94bd79bcb5c1080647731 consensus state has no leader
I20251029 08:52:10.755502 899 tablet_service.cc:1505] Processing CreateTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 (DEFAULT_TABLE table=default.Table_81ab02d14c514f1f9c7622f3776b4552 [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.755774 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbf9e36c47be4509a998be8e7138475b consensus state has no leader
I20251029 08:52:10.757015 901 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4e208427b94bd79bcb5c1080647731 (DEFAULT_TABLE table=default.Table_5b250161ee2248cd8188be4cb32cab6b [id=064c00240f9c48ceac41175a2b8306bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:10.757515 1177 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.004s sys 0.008s
I20251029 08:52:10.757557 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:10.757704 891 tablet_service.cc:1505] Processing CreateTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 (DEFAULT_TABLE table=default.Table_877007fc953a49b189bb45e236141e2d [id=d45481f7479f46d3bc3b014c7a437f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.758888 889 tablet_service.cc:1505] Processing CreateTablet for tablet 15dae38bcced496e90ca8eec280a8317 (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.759356 1051 tablet_service.cc:1505] Processing CreateTablet for tablet c32679cbb1884f09addb0f7c26ac941e (DEFAULT_TABLE table=default.Table_fb40280e4c7d421889ad47f81c4574b2 [id=a19e19f35785473e8533055ad792e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.747241 1183 consensus_queue.cc:260] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.760785 996 leader_election.cc:304] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:10.762343 1176 log.cc:1199] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:10.765393 1438 raft_consensus.cc:2804] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:10.771656 1177 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.706s user 0.004s sys 0.008s
I20251029 08:52:10.771978 1438 raft_consensus.cc:697] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:10.772332 1438 consensus_queue.cc:237] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.773523 1177 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:10.773550 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34729c3f97a44713a352bde297a80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.773680 1177 ts_tablet_manager.cc:1403] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.963s user 0.006s sys 0.012s
I20251029 08:52:10.774183 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d306be1d51674c2abc506d248e05a9c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.774212 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444a5d489434696ae178c6d650bf8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.774868 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62597189ecb047788b83788b514e9099. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.775105 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3222378209e141649a30f5c940ae2c23" } modify_external_catalogs: true
I20251029 08:52:10.775290 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3222378209e141649a30f5c940ae2c23" } modify_external_catalogs: true
I20251029 08:52:10.775470 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32679cbb1884f09addb0f7c26ac941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.775920 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f8c4f6e0aec748b9945c2dfe3cc5247a" } modify_external_catalogs: true
I20251029 08:52:10.776113 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f8c4f6e0aec748b9945c2dfe3cc5247a" } modify_external_catalogs: true
I20251029 08:52:10.776690 1183 ts_tablet_manager.cc:1434] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.190s user 0.001s sys 0.002s
I20251029 08:52:10.778548 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_69a638baaadc493f9439102862bb7607" } modify_external_catalogs: true
I20251029 08:52:10.778703 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_69a638baaadc493f9439102862bb7607" } modify_external_catalogs: true
I20251029 08:52:10.780061 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9fe3e9a507441b8ba5953c0120289c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.786639 753 tablet_service.cc:1505] Processing CreateTablet for tablet 471737aa35e94dcfaa2b3451cc8f2014 (DEFAULT_TABLE table=default.Table_81405043420f4711a665ddfcd6bb7d1b [id=427664ed0296404d9a435ff598f2d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.789321 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.789666 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bed64a27164baa8177b87b54b5947b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.790730 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c5d76d3bd24eca923ed08a72bbc56b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.793340 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_247e1528f531419d80295027ead897a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.795701 1183 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:10.800825 761 tablet_service.cc:1505] Processing CreateTablet for tablet 31bed64a27164baa8177b87b54b5947b (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.801611 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bed64a27164baa8177b87b54b5947b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.803547 1176 ts_tablet_manager.cc:1950] T 23b534aaa51b4fad9dc3fc915de947f1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:10.804411 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 346) took 1637 ms. Trace:
I20251029 08:52:10.806298 1176 rpcz_store.cc:276] 1029 08:52:09.167154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.167344 (+ 190us) service_pool.cc:225] Handling call
1029 08:52:10.365686 (+1198342us) ts_tablet_manager.cc:1151] Deleting tablet 23b534aaa51b4fad9dc3fc915de947f1
1029 08:52:10.365696 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.533523 (+167827us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.720412 (+186889us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.804374 (+ 83962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198174}
I20251029 08:52:10.806887 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 347) took 1639 ms. Trace:
I20251029 08:52:10.826511 763 tablet_service.cc:1505] Processing CreateTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 (DEFAULT_TABLE table=default.Table_81ab02d14c514f1f9c7622f3776b4552 [id=edda8e15b59b472886bb1eba8fec8739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.827862 756 tablet_service.cc:1505] Processing CreateTablet for tablet 15dae38bcced496e90ca8eec280a8317 (DEFAULT_TABLE table=default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.825453 1176 rpcz_store.cc:276] 1029 08:52:09.167652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.167830 (+ 178us) service_pool.cc:225] Handling call
1029 08:52:10.806743 (+1638913us) ts_tablet_manager.cc:1151] Deleting tablet de78ea608c904aeea7f7145b9c80f45b
1029 08:52:10.806753 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.806858 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638628}
I20251029 08:52:10.831828 1179 ts_tablet_manager.cc:1929] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:10.831988 1179 log.cc:1199] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c8cb0c6c6a1c4877a89476d4822f48db
I20251029 08:52:10.832558 1179 ts_tablet_manager.cc:1950] T c8cb0c6c6a1c4877a89476d4822f48db P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:10.832907 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 346) took 1628 ms. Trace:
I20251029 08:52:10.833022 1179 rpcz_store.cc:276] 1029 08:52:09.204269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.204405 (+ 136us) service_pool.cc:225] Handling call
1029 08:52:10.409126 (+1204721us) ts_tablet_manager.cc:1151] Deleting tablet c8cb0c6c6a1c4877a89476d4822f48db
1029 08:52:10.409137 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.563796 (+154659us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.830513 (+266717us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.832869 (+ 2356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166480}
I20251029 08:52:10.833648 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 347) took 1586 ms. Trace:
I20251029 08:52:10.833784 1179 rpcz_store.cc:276] 1029 08:52:09.247174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.247229 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:10.833482 (+1586253us) ts_tablet_manager.cc:1151] Deleting tablet 74fe974a368a4afe8a5da6dfd1036400
1029 08:52:10.833493 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.833614 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583828}
I20251029 08:52:10.834467 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 348) took 1568 ms. Trace:
I20251029 08:52:10.834708 1179 rpcz_store.cc:276] 1029 08:52:09.265833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.265950 (+ 117us) service_pool.cc:225] Handling call
1029 08:52:10.834236 (+1568286us) ts_tablet_manager.cc:1151] Deleting tablet ea13e9ea2c11479ea86e3fd7f2a9e59c
1029 08:52:10.834245 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.834398 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560298}
I20251029 08:52:10.835388 1179 tablet_replica.cc:333] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:10.838125 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 348) took 1660 ms. Trace:
I20251029 08:52:10.838640 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15dae38bcced496e90ca8eec280a8317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.839589 762 tablet_service.cc:1505] Processing CreateTablet for tablet 59c90087c9e94665a9b3fac798304af7 (DEFAULT_TABLE table=default.Table_266c99fef9414223bd1a5a73dbbf9bd5 [id=5b45ec162f5e43d4a7976bb7f6426dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.841120 760 tablet_service.cc:1505] Processing CreateTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 (DEFAULT_TABLE table=default.Table_877007fc953a49b189bb45e236141e2d [id=d45481f7479f46d3bc3b014c7a437f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.841427 1176 rpcz_store.cc:276] 1029 08:52:09.172248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.172477 (+ 229us) service_pool.cc:225] Handling call
1029 08:52:10.828627 (+1656150us) ts_tablet_manager.cc:1151] Deleting tablet a3ba512bd4f04bd9aceb4cb210fe9689
1029 08:52:10.831248 (+ 2621us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.832982 (+ 1734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655956}
I20251029 08:52:10.842008 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 349) took 1669 ms. Trace:
I20251029 08:52:10.842119 1176 rpcz_store.cc:276] 1029 08:52:09.172836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.173051 (+ 215us) service_pool.cc:225] Handling call
1029 08:52:10.841860 (+1668809us) ts_tablet_manager.cc:1151] Deleting tablet 4e2773cae9814df79ee60ea8b0fe4e9d
1029 08:52:10.841869 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.841975 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668567}
I20251029 08:52:10.842602 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 357) took 1554 ms. Trace:
I20251029 08:52:10.842813 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26ff78aa5f94ae49a197daf2be00aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.842809 1707 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 08:52:10.843163 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ab56045bd814d59831a98618c759af4 consensus state has no leader
I20251029 08:52:10.843662 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_43e66e8dbdf240a1b5eb4233ece857c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.844832 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_375cce62abe84482b107057a691c49f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.845626 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9fe3e9a507441b8ba5953c0120289c3 consensus state has no leader
I20251029 08:52:10.846002 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cfc355ba254a4851867ed2eed4f56cd3" } new_table_name: "default.Table_2576901b40a44a1a892f8bf476ee9519" modify_external_catalogs: true
I20251029 08:52:10.850517 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bed64a27164baa8177b87b54b5947b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.850654 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.851352 747 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4e208427b94bd79bcb5c1080647731 (DEFAULT_TABLE table=default.Table_5b250161ee2248cd8188be4cb32cab6b [id=064c00240f9c48ceac41175a2b8306bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.851856 759 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.852205 759 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.842705 1176 rpcz_store.cc:276] 1029 08:52:09.290810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.292084 (+ 1274us) service_pool.cc:225] Handling call
1029 08:52:10.842477 (+1550393us) ts_tablet_manager.cc:1151] Deleting tablet 2723cdd513ee4e8ca6773b38f785b09e
1029 08:52:10.842485 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.842573 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550126}
I20251029 08:52:10.859093 589 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 220400be898846738d711d4eff78344d
I20251029 08:52:10.859536 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e747c0eb07df4d70925a7232b448b39e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.860469 759 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.860800 759 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.861429 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b60ce497df09480daa025d919c6509e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:10.863653 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_325e6358252a40b0bdd59c1a0664ed7e" } modify_external_catalogs: true
I20251029 08:52:10.863866 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_325e6358252a40b0bdd59c1a0664ed7e" } modify_external_catalogs: true
I20251029 08:52:10.867209 1068 raft_consensus.cc:3060] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:10.869882 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 358) took 1577 ms. Trace:
I20251029 08:52:10.870952 1176 rpcz_store.cc:276] 1029 08:52:09.293805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.295203 (+ 1398us) service_pool.cc:225] Handling call
1029 08:52:10.869682 (+1574479us) ts_tablet_manager.cc:1151] Deleting tablet 215cd666c71a4179a19e298868a88e96
1029 08:52:10.869714 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.869844 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563378}
I20251029 08:52:10.871518 1176 tablet_replica.cc:333] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:10.871609 1179 raft_consensus.cc:2243] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:10.871829 1179 raft_consensus.cc:2272] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.872224 1176 raft_consensus.cc:2243] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:10.872437 1176 raft_consensus.cc:2272] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:10.874141 1704 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251029 08:52:10.875303 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a506e35afe6a4c87901f27c434f3b348 consensus state has no leader
I20251029 08:52:10.875761 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_064ffeda7eaa4da6854b9395b0c98345" } modify_external_catalogs: true
I20251029 08:52:10.875979 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_064ffeda7eaa4da6854b9395b0c98345" } modify_external_catalogs: true
I20251029 08:52:10.880167 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15dae38bcced496e90ca8eec280a8317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.883759 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4e208427b94bd79bcb5c1080647731. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.883922 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4e208427b94bd79bcb5c1080647731. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.885205 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c90087c9e94665a9b3fac798304af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.885959 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9fe3e9a507441b8ba5953c0120289c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.887399 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471737aa35e94dcfaa2b3451cc8f2014. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.890357 1176 ts_tablet_manager.cc:1916] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.891459 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4e208427b94bd79bcb5c1080647731. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.891796 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c90087c9e94665a9b3fac798304af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.891836 1179 ts_tablet_manager.cc:1916] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:10.892125 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bb0fd4c62d47a2b10719ebd55851c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.902513 1183 tablet_bootstrap.cc:654] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:10.907634 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26ff78aa5f94ae49a197daf2be00aa5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:10.916765 861 consensus_peers.cc:597] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8cb0c6c6a1c4877a89476d4822f48db. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:10.861495 759 tablet_service.cc:1505] Processing CreateTablet for tablet a506e35afe6a4c87901f27c434f3b348 (DEFAULT_TABLE table=default.Table_cfc355ba254a4851867ed2eed4f56cd3 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.922500 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.807107 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 23b534aaa51b4fad9dc3fc915de947f1 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
W20251029 08:52:10.922976 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43e66e8dbdf240a1b5eb4233ece857c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.923084 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 115953us
I20251029 08:52:10.923971 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
W20251029 08:52:10.924265 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 74fe974a368a4afe8a5da6dfd1036400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fe974a368a4afe8a5da6dfd1036400 already in progress: creating tablet
W20251029 08:52:10.924342 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e747c0eb07df4d70925a7232b448b39e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.924510 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea13e9ea2c11479ea86e3fd7f2a9e59c already in progress: creating tablet
W20251029 08:52:10.925345 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b60ce497df09480daa025d919c6509e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:10.925496 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet de78ea608c904aeea7f7145b9c80f45b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de78ea608c904aeea7f7145b9c80f45b already in progress: creating tablet
W20251029 08:52:10.925782 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a3ba512bd4f04bd9aceb4cb210fe9689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ba512bd4f04bd9aceb4cb210fe9689 already in progress: creating tablet
W20251029 08:52:10.925963 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4e2773cae9814df79ee60ea8b0fe4e9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2773cae9814df79ee60ea8b0fe4e9d already in progress: creating tablet
W20251029 08:52:10.926229 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2723cdd513ee4e8ca6773b38f785b09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2723cdd513ee4e8ca6773b38f785b09e already in progress: creating tablet
W20251029 08:52:10.926450 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 215cd666c71a4179a19e298868a88e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215cd666c71a4179a19e298868a88e96 already in progress: creating tablet
W20251029 08:52:10.929785 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a506e35afe6a4c87901f27c434f3b348 (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet a506e35afe6a4c87901f27c434f3b348 consensus state has no leader
W20251029 08:52:10.930418 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_375cce62abe84482b107057a691c49f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.934176 755 tablet_service.cc:1505] Processing CreateTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 (DEFAULT_TABLE table=default.Table_11a78e5b37424289a184f8bccbd263d4 [id=c54d460a25104cc19f43a5517c203b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.925927 903 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.938897 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_25ceffa901324b3e85b4d8e8cd7268a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:10.939510 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25ceffa901324b3e85b4d8e8cd7268a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:10.940903 751 tablet_service.cc:1505] Processing CreateTablet for tablet d444a5d489434696ae178c6d650bf8d5 (DEFAULT_TABLE table=default.Table_55491b160d0e4e6185be2e0fee2e67a5 [id=8431eac1a49a41c080964575e16345d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.945477 894 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:10.945672 896 consensus_meta.cc:287] T af913080bfe44a86a76704ffa32cd9e1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent flushing consensus metadata: real 0.622s user 0.003s sys 0.000s
I20251029 08:52:10.946902 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:10.947580 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bb0fd4c62d47a2b10719ebd55851c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.940764 752 tablet_service.cc:1505] Processing CreateTablet for tablet 62597189ecb047788b83788b514e9099 (DEFAULT_TABLE table=default.Table_31cc9d5473cd45dcb2a4fb9609ff132b [id=c14f386ee208443db44d014c6b613070]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.949155 903 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:10.949581 903 tablet_service.cc:1505] Processing CreateTablet for tablet d444a5d489434696ae178c6d650bf8d5 (DEFAULT_TABLE table=default.Table_55491b160d0e4e6185be2e0fee2e67a5 [id=8431eac1a49a41c080964575e16345d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.947059 1177 raft_consensus.cc:359] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.951097 745 tablet_service.cc:1505] Processing CreateTablet for tablet 34729c3f97a44713a352bde297a80027 (DEFAULT_TABLE table=default.Table_15cd2138e80e429db3004528a89ec2bf [id=a8a2e644aa73491b9d22e81babf2e582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.952284 764 tablet_service.cc:1505] Processing CreateTablet for tablet c32679cbb1884f09addb0f7c26ac941e (DEFAULT_TABLE table=default.Table_fb40280e4c7d421889ad47f81c4574b2 [id=a19e19f35785473e8533055ad792e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.953598 1707 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:10.958364 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62597189ecb047788b83788b514e9099. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.958770 1177 raft_consensus.cc:385] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:10.960749 897 tablet_service.cc:1505] Processing CreateTablet for tablet c32679cbb1884f09addb0f7c26ac941e (DEFAULT_TABLE table=default.Table_fb40280e4c7d421889ad47f81c4574b2 [id=a19e19f35785473e8533055ad792e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.959901 896 tablet_service.cc:1505] Processing CreateTablet for tablet 34729c3f97a44713a352bde297a80027 (DEFAULT_TABLE table=default.Table_15cd2138e80e429db3004528a89ec2bf [id=a8a2e644aa73491b9d22e81babf2e582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.962747 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444a5d489434696ae178c6d650bf8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.963624 894 tablet_service.cc:1505] Processing CreateTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 (DEFAULT_TABLE table=default.Table_11a78e5b37424289a184f8bccbd263d4 [id=c54d460a25104cc19f43a5517c203b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.965750 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_971780f9a8d5435a9f6a94d5727f5119" } new_table_name: "default.Table_175a44baba964b30b0fd67d4cd921479" modify_external_catalogs: true
I20251029 08:52:10.967394 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_abf2ca58419041dbb83bfa81e6da8f08" } new_table_name: "default.Table_5a8eb1fd4c9c425ca97d2fb315b7e555" modify_external_catalogs: true
I20251029 08:52:10.969491 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32679cbb1884f09addb0f7c26ac941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.972373 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444a5d489434696ae178c6d650bf8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.972939 898 tablet_service.cc:1505] Processing CreateTablet for tablet a506e35afe6a4c87901f27c434f3b348 (DEFAULT_TABLE table=default.Table_cfc355ba254a4851867ed2eed4f56cd3 [id=40d62f7b6ed74484a47a175de461b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:10.976879 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bb0fd4c62d47a2b10719ebd55851c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.981443 1177 raft_consensus.cc:740] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:10.982623 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32679cbb1884f09addb0f7c26ac941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:10.987599 1655 raft_consensus.cc:493] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:10.991813 1177 consensus_queue.cc:260] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.993166 1175 ts_tablet_manager.cc:1929] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:10.995095 1691 raft_consensus.cc:493] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:10.995234 1691 raft_consensus.cc:515] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:10.995811 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34729c3f97a44713a352bde297a80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.001113 1175 log.cc:1199] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/61ff3274c7104c73bd35768c229833b1
I20251029 08:52:11.002070 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506e35afe6a4c87901f27c434f3b348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.008118 1177 ts_tablet_manager.cc:1434] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.222s user 0.003s sys 0.000s
I20251029 08:52:11.009102 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet be706771a95e45e5a20a15614eadfe1a (table default.Table_a69aba63f41c4596a1017f38f6ed6532 [id=e20293dc36d54acf9ee6f9bf7ac55834]). Expected version 1 got 0
I20251029 08:52:11.009328 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_d378e85a66bb4e40b69d49ec1cf86a51 [id=334d077ad2634b9490979d41a64e7630]). Expected version 6 got 5
I20251029 08:52:11.009548 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]). Expected version 1 got 0
I20251029 08:52:11.010519 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7fdbe6681894ffa8264cdfa486f16e6" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:11.010684 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b777b26318974b9f92919063554e2100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.011206 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b777b26318974b9f92919063554e2100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.012905 1691 leader_election.cc:290] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:11.013867 861 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.014024 1177 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:11.019176 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24834985a71349b9865840e9b5537d72 consensus state has no leader
I20251029 08:52:11.019644 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6344489711f94203b5eae7cf63255c61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.020257 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6344489711f94203b5eae7cf63255c61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:11.021487 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10baa0547180493f872a23614cf4c608 consensus state has no leader
I20251029 08:52:11.021773 596 master_service.cc:954] ReplaceTablet: received request to replace tablet e8c4d793f10147fba5e4f44474575bf0 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.031224 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 43c3074caee74069a8a7f1b87721d726
I20251029 08:52:11.031749 597 catalog_manager.cc:6719] ReplaceTablet: tablet 43c3074caee74069a8a7f1b87721d726 deleted and replaced by tablet 527a83465a844849811b0f9d26428af0
I20251029 08:52:11.032932 1179 ts_tablet_manager.cc:1929] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:11.033090 1179 log.cc:1199] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4b55ae84c8d441459a5931abda471f6c
I20251029 08:52:11.033551 1179 ts_tablet_manager.cc:1950] T 4b55ae84c8d441459a5931abda471f6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:11.033879 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 349) took 1758 ms. Trace:
I20251029 08:52:11.033968 1179 rpcz_store.cc:276] 1029 08:52:09.274951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.275066 (+ 115us) service_pool.cc:225] Handling call
1029 08:52:10.835300 (+1560234us) ts_tablet_manager.cc:1151] Deleting tablet 4b55ae84c8d441459a5931abda471f6c
1029 08:52:10.835309 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.999626 (+164317us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.032773 (+ 33147us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.033831 (+ 1058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1558928}
I20251029 08:52:11.034350 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34729c3f97a44713a352bde297a80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.037144 1183 tablet_bootstrap.cc:492] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:11.037278 1183 ts_tablet_manager.cc:1403] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.260s user 0.011s sys 0.003s
I20251029 08:52:11.038456 1183 raft_consensus.cc:359] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.040454 1688 raft_consensus.cc:493] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:11.040628 1688 raft_consensus.cc:515] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.042189 1688 leader_election.cc:290] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:11.043135 1655 raft_consensus.cc:515] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.043773 1655 leader_election.cc:290] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:11.045470 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9907d044daa145168417371c2bd66544
I20251029 08:52:11.046320 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:11.046937 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:11.047621 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e3c24c0d32f74435bd9d28f2415e2c47
I20251029 08:52:11.049494 1178 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.049531 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:11.049893 596 catalog_manager.cc:6719] ReplaceTablet: tablet e8c4d793f10147fba5e4f44474575bf0 deleted and replaced by tablet 85a806cc04e641d9a1c4af97a24b13de
I20251029 08:52:11.052345 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d45106f234434f44ade17ce063aeacbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.052953 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45106f234434f44ade17ce063aeacbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.054193 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f159f31991334c3cac52a8b4818345e8" } new_table_name: "default.Table_5e62c5fdfba04b7287b6ec49e5af68d1" modify_external_catalogs: true
I20251029 08:52:11.055027 1178 ts_tablet_manager.cc:1403] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.622s user 0.005s sys 0.009s
I20251029 08:52:11.056529 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5bd680f724ec45d7816addcd2a67ce84" } new_table_name: "default.Table_06bcd895f6814a55a1dbda4c7b9e1acc" modify_external_catalogs: true
I20251029 08:52:11.061040 1183 raft_consensus.cc:385] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.061182 1183 raft_consensus.cc:740] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.061060 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.061455 1183 consensus_queue.cc:260] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.062005 1183 ts_tablet_manager.cc:1434] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251029 08:52:11.062299 1183 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:11.063071 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.047219 1707 tablet.cc:1722] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:11.053905 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 57159c264d6641afa3415d33c1d32efe (DEFAULT_TABLE table=default.Table_344943c2bd4d4ea1bf1095ed952a2dd2 [id=7a5c0a7295414daa823ea0df7d47764d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.065044 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_04d72f6fb117444a96f5d646217085e2" } modify_external_catalogs: true
I20251029 08:52:11.065238 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_04d72f6fb117444a96f5d646217085e2" } modify_external_catalogs: true
I20251029 08:52:11.065446 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57159c264d6641afa3415d33c1d32efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.054476 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.055006 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.055461 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.055953 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.074617 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_007a17f442c74a71a6bb3775642f35fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.075331 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_007a17f442c74a71a6bb3775642f35fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.076643 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_183dac6abe3b4b6d8ef9614f941af3a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.077209 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_183dac6abe3b4b6d8ef9614f941af3a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.077811 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.079106 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.079118 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.079169 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3d67d4e4f61646fe8e8864734badbc5d" } new_table_name: "default.Table_6b5e334f84e049eea474e2587d954f23" modify_external_catalogs: true
I20251029 08:52:11.082031 1175 ts_tablet_manager.cc:1950] T 61ff3274c7104c73bd35768c229833b1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:11.084615 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 356) took 1700 ms. Trace:
I20251029 08:52:11.084720 1179 rpcz_store.cc:276] 1029 08:52:09.383823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.442430 (+ 58607us) service_pool.cc:225] Handling call
1029 08:52:11.084242 (+1641812us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:11.084253 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.084362 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638728}
I20251029 08:52:11.087226 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_d378e85a66bb4e40b69d49ec1cf86a51 [id=334d077ad2634b9490979d41a64e7630]). Expected version 6 got 5
I20251029 08:52:11.078648 1717 tablet.cc:1722] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:11.087466 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]). Expected version 3 got 0
W20251029 08:52:11.088117 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85b300f17a704f7dba1aa3724fcc2de7 consensus state has no leader
I20251029 08:52:11.088281 1177 tablet_bootstrap.cc:654] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.088574 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 351047c725f8488abbd074bd40790b48
I20251029 08:52:11.088691 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]) successfully deleted
W20251029 08:52:11.089008 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c66cbb495f5142d79a674f7d5adc505e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c66cbb495f5142d79a674f7d5adc505e already in progress: creating tablet
W20251029 08:52:11.089255 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bda3e143699b41b6bba308c83bbb3e0d consensus state has no leader
I20251029 08:52:11.090579 597 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98] alter complete (version 3)
I20251029 08:52:11.091122 596 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]). Expected version 1 got 0
I20251029 08:52:11.096755 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.089828 1710 tablet.cc:1722] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:11.100548 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.100775 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 (DEFAULT_TABLE table=default.Table_5df6707d7a754de3b61f32cf68786b2e [id=2f663ee400904f1d9e21d15b83c58e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.101163 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c9946f3f14a2aab11f4cdaa70a5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.094962 904 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.101779 887 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.101929 892 tablet_service.cc:1505] Processing CreateTablet for tablet 57159c264d6641afa3415d33c1d32efe (DEFAULT_TABLE table=default.Table_344943c2bd4d4ea1bf1095ed952a2dd2 [id=7a5c0a7295414daa823ea0df7d47764d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:11.102106 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0ff0658e0274cf49c389ad602d91b03 consensus state has no leader
I20251029 08:52:11.106467 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 19fdc381a0d949b0b93cef399daa3021 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.108095 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cdf03acb92449d847aec897868b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.108152 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 58932f38c0d64aec9a02238eaea56a25 (DEFAULT_TABLE table=default.Table_2d05ded03f9d4415ada974d91d47ec0c [id=60ed38dd0d6e4bc5851de7c04b92abfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.108603 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58932f38c0d64aec9a02238eaea56a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.109027 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 365) took 1562 ms. Trace:
I20251029 08:52:11.109141 1179 rpcz_store.cc:276] 1029 08:52:09.522744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.522839 (+ 95us) service_pool.cc:225] Handling call
1029 08:52:11.085100 (+1562261us) ts_tablet_manager.cc:1151] Deleting tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
1029 08:52:11.085108 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.085212 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562000}
I20251029 08:52:11.109812 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 366) took 1586 ms. Trace:
I20251029 08:52:11.109918 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 333) took 2970 ms. Trace:
I20251029 08:52:11.109916 1179 rpcz_store.cc:276] 1029 08:52:09.523144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.523210 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:11.109606 (+1586396us) ts_tablet_manager.cc:1151] Deleting tablet 5385fef404a4463595074f0c8ff822d7
1029 08:52:11.109617 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.109747 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586213}
I20251029 08:52:11.110476 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 367) took 1585 ms. Trace:
I20251029 08:52:11.110575 1179 rpcz_store.cc:276] 1029 08:52:09.524610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.529025 (+ 4415us) service_pool.cc:225] Handling call
1029 08:52:11.110283 (+1581258us) ts_tablet_manager.cc:1151] Deleting tablet 2fb380018f46465e82ac6223ecf8dd0d
1029 08:52:11.110292 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.110415 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574996}
I20251029 08:52:11.111399 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 368) took 1577 ms. Trace:
I20251029 08:52:11.111506 1179 rpcz_store.cc:276] 1029 08:52:09.534086 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.534843 (+ 757us) service_pool.cc:225] Handling call
1029 08:52:11.111221 (+1576378us) ts_tablet_manager.cc:1151] Deleting tablet bc96ef1a94844a32923087c44a5bcfc8
1029 08:52:11.111233 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.111334 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566856}
I20251029 08:52:11.112071 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 375) took 1423 ms. Trace:
I20251029 08:52:11.112175 1179 rpcz_store.cc:276] 1029 08:52:09.688279 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.692068 (+ 3789us) service_pool.cc:225] Handling call
1029 08:52:11.111887 (+1419819us) ts_tablet_manager.cc:1151] Deleting tablet 8991ea78321545be85d5c480841e4751
1029 08:52:11.111899 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.111989 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419587}
I20251029 08:52:11.112751 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 400) took 1025 ms. Trace:
I20251029 08:52:11.112843 1179 rpcz_store.cc:276] 1029 08:52:10.087469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.088168 (+ 699us) service_pool.cc:225] Handling call
1029 08:52:11.112566 (+1024398us) ts_tablet_manager.cc:1151] Deleting tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
1029 08:52:11.112581 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.112686 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024092}
I20251029 08:52:11.116436 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet a20b64ff177241309e5d59e84f0bafa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.119123 891 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.121712 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:11.121748 895 tablet_service.cc:1505] Processing CreateTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 (DEFAULT_TABLE table=default.Table_c407207334e644189fdbfcae1dc12ba9 [id=e1fb342ea0a541ce9f668a9410db6487]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.122052 902 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.122136 589 catalog_manager.cc:6719] ReplaceTablet: tablet 19fdc381a0d949b0b93cef399daa3021 deleted and replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033
I20251029 08:52:11.122519 888 tablet_service.cc:1505] Processing CreateTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 (DEFAULT_TABLE table=default.Table_5df6707d7a754de3b61f32cf68786b2e [id=2f663ee400904f1d9e21d15b83c58e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.122597 1175 rpcz_store.cc:276] 1029 08:52:08.123352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.123579 (+ 227us) service_pool.cc:225] Handling call
1029 08:52:10.615406 (+2491827us) ts_tablet_manager.cc:1151] Deleting tablet 61ff3274c7104c73bd35768c229833b1
1029 08:52:10.615417 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:10.700309 (+ 84892us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:10.969301 (+268992us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.093944 (+124643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421789}
I20251029 08:52:11.122540 890 tablet_service.cc:1505] Processing CreateTablet for tablet 58932f38c0d64aec9a02238eaea56a25 (DEFAULT_TABLE table=default.Table_2d05ded03f9d4415ada974d91d47ec0c [id=60ed38dd0d6e4bc5851de7c04b92abfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.126253 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506e35afe6a4c87901f27c434f3b348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.126401 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:11.128582 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 139609us
I20251029 08:52:11.128675 1183 tablet_bootstrap.cc:654] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.129289 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be706771a95e45e5a20a15614eadfe1a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:11.129801 1178 raft_consensus.cc:359] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.130017 1178 raft_consensus.cc:385] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.130120 1178 raft_consensus.cc:740] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.130723 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.131994 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 (DEFAULT_TABLE table=default.Table_c407207334e644189fdbfcae1dc12ba9 [id=e1fb342ea0a541ce9f668a9410db6487]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.132795 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9499d0099d1c4edc843fb8d29ca3b1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.135247 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 338) took 2766 ms. Trace:
W20251029 08:52:11.130702 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa0ff14e19b4ad7a8b33796481f0f3f already in progress: creating tablet
W20251029 08:52:11.136408 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5385fef404a4463595074f0c8ff822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5385fef404a4463595074f0c8ff822d7 already in progress: creating tablet
W20251029 08:52:11.136629 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2fb380018f46465e82ac6223ecf8dd0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb380018f46465e82ac6223ecf8dd0d already in progress: creating tablet
W20251029 08:52:11.136889 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bc96ef1a94844a32923087c44a5bcfc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc96ef1a94844a32923087c44a5bcfc8 already in progress: creating tablet
W20251029 08:52:11.137142 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8991ea78321545be85d5c480841e4751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8991ea78321545be85d5c480841e4751 already in progress: creating tablet
W20251029 08:52:11.137337 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d already in progress: creating tablet
I20251029 08:52:11.137518 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:11.135970 1175 rpcz_store.cc:276] 1029 08:52:08.368751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.368878 (+ 127us) service_pool.cc:225] Handling call
1029 08:52:11.122959 (+2754081us) ts_tablet_manager.cc:1151] Deleting tablet 4deeec492a2545ca86583c6ea899a0ab
1029 08:52:11.131384 (+ 8425us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.134944 (+ 3560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705035}
W20251029 08:52:11.138239 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4deeec492a2545ca86583c6ea899a0ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4deeec492a2545ca86583c6ea899a0ab already in progress: creating tablet
I20251029 08:52:11.138325 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 398) took 1145 ms. Trace:
I20251029 08:52:11.138413 1179 rpcz_store.cc:276] 1029 08:52:09.992371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.010059 (+ 17688us) service_pool.cc:225] Handling call
1029 08:52:11.137923 (+1127864us) ts_tablet_manager.cc:1151] Deleting tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
1029 08:52:11.137936 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.138139 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038805}
I20251029 08:52:11.139828 886 tablet_service.cc:1505] Processing CreateTablet for tablet 62597189ecb047788b83788b514e9099 (DEFAULT_TABLE table=default.Table_31cc9d5473cd45dcb2a4fb9609ff132b [id=c14f386ee208443db44d014c6b613070]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.140358 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 401) took 1052 ms. Trace:
I20251029 08:52:11.140440 1179 rpcz_store.cc:276] 1029 08:52:10.087558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.088026 (+ 468us) service_pool.cc:225] Handling call
1029 08:52:11.140173 (+1052147us) ts_tablet_manager.cc:1151] Deleting tablet bf29e030892d44e1ad4d1ca4f5aeab3b
1029 08:52:11.140182 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.140304 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027888}
I20251029 08:52:11.141040 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 402) took 1053 ms. Trace:
W20251029 08:52:11.141279 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 already in progress: creating tablet
W20251029 08:52:11.141614 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bf29e030892d44e1ad4d1ca4f5aeab3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf29e030892d44e1ad4d1ca4f5aeab3b already in progress: creating tablet
I20251029 08:52:11.141762 1179 rpcz_store.cc:276] 1029 08:52:10.087711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.089004 (+ 1293us) service_pool.cc:225] Handling call
1029 08:52:11.140886 (+1051882us) ts_tablet_manager.cc:1151] Deleting tablet 14b665b63a7341d699d499ffc8cfd9f7
1029 08:52:11.140898 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.140992 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028323}
I20251029 08:52:11.142223 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 339) took 2748 ms. Trace:
I20251029 08:52:11.144569 1175 rpcz_store.cc:276] 1029 08:52:08.392333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.392928 (+ 595us) service_pool.cc:225] Handling call
1029 08:52:11.137894 (+2744966us) ts_tablet_manager.cc:1151] Deleting tablet 2e96cbdc4b7845b3823617a8e9a8c8b2
1029 08:52:11.138766 (+ 872us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.140545 (+ 1779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2719674}
I20251029 08:52:11.144861 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 399) took 1057 ms. Trace:
I20251029 08:52:11.144969 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 344) took 2471 ms. Trace:
I20251029 08:52:11.144953 1179 rpcz_store.cc:276] 1029 08:52:10.087354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.089137 (+ 1783us) service_pool.cc:225] Handling call
1029 08:52:11.144701 (+1055564us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:11.144714 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.144810 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031906}
I20251029 08:52:11.145112 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ee85eba8daea413bb0586c2a113351a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.144804 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e857915b93d94731a6672caf050b7807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.145668 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee85eba8daea413bb0586c2a113351a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:11.146389 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 already in progress: creating tablet
I20251029 08:52:11.147039 1175 rpcz_store.cc:276] 1029 08:52:08.673794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.673853 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:11.144825 (+2470972us) ts_tablet_manager.cc:1151] Deleting tablet 1ae561a995c94fadb288b6395c55a1bf
1029 08:52:11.144836 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.144924 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470753}
W20251029 08:52:11.145674 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e857915b93d94731a6672caf050b7807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.147439 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 345) took 2430 ms. Trace:
I20251029 08:52:11.147635 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6eaf722368b14c059665538870edd00f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.148344 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eaf722368b14c059665538870edd00f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.148380 596 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
W20251029 08:52:11.148595 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 1ae561a995c94fadb288b6395c55a1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae561a995c94fadb288b6395c55a1bf already in progress: creating tablet
I20251029 08:52:11.149008 1175 rpcz_store.cc:276] 1029 08:52:08.717136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.717187 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:11.147297 (+2430110us) ts_tablet_manager.cc:1151] Deleting tablet e4f80728a7a04583bada485707beb806
1029 08:52:11.147308 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.147410 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2429882}
I20251029 08:52:11.149412 1175 tablet_replica.cc:333] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:11.150367 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 14b665b63a7341d699d499ffc8cfd9f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b665b63a7341d699d499ffc8cfd9f7 already in progress: creating tablet
W20251029 08:52:11.150651 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet eed66c3fa3f848df905a2a31fd553474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eed66c3fa3f848df905a2a31fd553474 already in progress: creating tablet
I20251029 08:52:11.151094 1179 tablet_replica.cc:333] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:11.151362 1179 raft_consensus.cc:2243] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.151957 1179 raft_consensus.cc:2272] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.154184 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:11.155073 1178 consensus_queue.cc:260] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.155802 1704 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 08:52:11.157059 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 09759a50a82f4b758449da9fbbc1f72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.149634 1175 raft_consensus.cc:2243] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:11.157127 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e4f80728a7a04583bada485707beb806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4f80728a7a04583bada485707beb806 already in progress: creating tablet
I20251029 08:52:11.157819 1175 raft_consensus.cc:2272] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.158453 1175 ts_tablet_manager.cc:1916] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:11.161001 720 leader_election.cc:343] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.161324 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6] alter complete (version 1)
I20251029 08:52:11.165784 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57159c264d6641afa3415d33c1d32efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.168206 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9499d0099d1c4edc843fb8d29ca3b1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.172216 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58932f38c0d64aec9a02238eaea56a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.173103 1176 ts_tablet_manager.cc:1929] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:11.175384 1175 ts_tablet_manager.cc:1929] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.180626 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:11.186650 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 133551us
I20251029 08:52:11.189615 761 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.192317 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.193502 761 tablet_service.cc:1505] Processing CreateTablet for tablet 57159c264d6641afa3415d33c1d32efe (DEFAULT_TABLE table=default.Table_344943c2bd4d4ea1bf1095ed952a2dd2 [id=7a5c0a7295414daa823ea0df7d47764d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.193928 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57159c264d6641afa3415d33c1d32efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.202313 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.211879 1704 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:39982 (request call id 419) took 1173 ms. Trace:
I20251029 08:52:11.212050 1704 rpcz_store.cc:276] 1029 08:52:10.038698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.170028 (+131330us) service_pool.cc:225] Handling call
1029 08:52:11.211841 (+1041813us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:10.170609 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 08:52:10.170726 (+ 117us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 08:52:10.170735 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
1029 08:52:10.170741 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 08:52:10.170783 (+ 42us) alter_schema_op.cc:128] START. Timestamp: 1761727930170737
1029 08:52:10.170789 (+ 6us) op_driver.cc:348] REPLICATION: starting
1029 08:52:10.170974 (+ 185us) log.cc:844] Serialized 173 byte log entry
1029 08:52:11.153369 (+982395us) op_driver.cc:464] REPLICATION: finished
1029 08:52:11.153595 (+ 226us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 08:52:11.156224 (+ 2629us) log.cc:844] Serialized 5 byte log entry
1029 08:52:11.208009 (+ 51785us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.209926 (+ 1917us) log.cc:844] Serialized 14 byte log entry
1029 08:52:11.211810 (+ 1884us) alter_schema_op.cc:73] Released schema lock
1029 08:52:11.211815 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":178,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":58686,"prepare.run_wall_time_us":217128,"raft.queue_time_us":46,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":240,"replication_time_us":982548,"thread_start_us":216119,"threads_started":1,"wal-append.queue_time_us":355348}]]}
I20251029 08:52:11.176146 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c9946f3f14a2aab11f4cdaa70a5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.219359 747 tablet_service.cc:1505] Processing CreateTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 (DEFAULT_TABLE table=default.Table_c407207334e644189fdbfcae1dc12ba9 [id=e1fb342ea0a541ce9f668a9410db6487]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.219676 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62597189ecb047788b83788b514e9099. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.178052 861 consensus_peers.cc:597] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b55ae84c8d441459a5931abda471f6c. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:11.219789 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9499d0099d1c4edc843fb8d29ca3b1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.182107 1177 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:11.224311 1177 ts_tablet_manager.cc:1403] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.216s user 0.004s sys 0.004s
I20251029 08:52:11.224331 752 tablet_service.cc:1505] Processing CreateTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 (DEFAULT_TABLE table=default.Table_5df6707d7a754de3b61f32cf68786b2e [id=2f663ee400904f1d9e21d15b83c58e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.224354 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 42de32d87ebb451090e0310e36e64277 (DEFAULT_TABLE table=default.Table_007a17f442c74a71a6bb3775642f35fd [id=3ca17547257d4c358a173ebef3ef4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.224445 895 tablet_service.cc:1505] Processing CreateTablet for tablet 42de32d87ebb451090e0310e36e64277 (DEFAULT_TABLE table=default.Table_007a17f442c74a71a6bb3775642f35fd [id=3ca17547257d4c358a173ebef3ef4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.224803 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42de32d87ebb451090e0310e36e64277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.225284 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c9946f3f14a2aab11f4cdaa70a5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.225239 1177 raft_consensus.cc:359] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.225446 1177 raft_consensus.cc:385] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.225528 1177 raft_consensus.cc:740] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.225773 1177 consensus_queue.cc:260] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.225972 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_611d65e05ca149d8910e6f16f99dce9c" } new_table_name: "default.Table_0073b1df0b7d4647bff7d80fe246edf6" modify_external_catalogs: true
I20251029 08:52:11.227537 1177 ts_tablet_manager.cc:1434] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:11.227869 1177 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:11.190163 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be706771a95e45e5a20a15614eadfe1a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:11.190351 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:11.228160 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.228400 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.228811 764 tablet_service.cc:1505] Processing CreateTablet for tablet 58932f38c0d64aec9a02238eaea56a25 (DEFAULT_TABLE table=default.Table_2d05ded03f9d4415ada974d91d47ec0c [id=60ed38dd0d6e4bc5851de7c04b92abfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.229256 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58932f38c0d64aec9a02238eaea56a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.229234 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d6e32e655391437ea03d14eade0cc35f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.229794 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6e32e655391437ea03d14eade0cc35f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.231393 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6f5c6155dbd84d46bccdc4c8679532b0" } new_table_name: "default.Table_5c0b5dd5fcc24859812282551226c27e" modify_external_catalogs: true
I20251029 08:52:11.235617 1176 log.cc:1199] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c8cb0c6c6a1c4877a89476d4822f48db
I20251029 08:52:11.235728 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_d378e85a66bb4e40b69d49ec1cf86a51 [id=334d077ad2634b9490979d41a64e7630] alter complete (version 6)
I20251029 08:52:11.193858 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:11.195922 1710 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 08:52:11.236827 1176 ts_tablet_manager.cc:1950] T c8cb0c6c6a1c4877a89476d4822f48db P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.237273 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 361) took 1932 ms. Trace:
I20251029 08:52:11.237389 1176 rpcz_store.cc:276] 1029 08:52:09.306391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.308049 (+ 1658us) service_pool.cc:225] Handling call
1029 08:52:10.871452 (+1563403us) ts_tablet_manager.cc:1151] Deleting tablet c8cb0c6c6a1c4877a89476d4822f48db
1029 08:52:10.871462 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.073320 (+201858us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.172967 (+ 99647us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.237221 (+ 64254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555050}
I20251029 08:52:11.238052 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 362) took 1921 ms. Trace:
I20251029 08:52:11.238272 1176 rpcz_store.cc:276] 1029 08:52:09.316324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.316455 (+ 131us) service_pool.cc:225] Handling call
1029 08:52:11.237834 (+1921379us) ts_tablet_manager.cc:1151] Deleting tablet 74fe974a368a4afe8a5da6dfd1036400
1029 08:52:11.237845 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.238000 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789191}
I20251029 08:52:11.238759 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 360) took 1936 ms. Trace:
I20251029 08:52:11.238902 1176 rpcz_store.cc:276] 1029 08:52:09.302493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.306307 (+ 3814us) service_pool.cc:225] Handling call
1029 08:52:11.238600 (+1932293us) ts_tablet_manager.cc:1151] Deleting tablet bd0c43367a8c47f390cc61d2764d61e4
1029 08:52:11.238610 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.238708 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787577}
I20251029 08:52:11.239256 1176 tablet_replica.cc:333] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.239507 1176 raft_consensus.cc:2243] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.239768 1176 raft_consensus.cc:2272] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.240541 1176 ts_tablet_manager.cc:1916] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.241250 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 308da528313044929b66040d6659f774 (DEFAULT_TABLE table=default.Table_6344489711f94203b5eae7cf63255c61 [id=205b9176bb44494b9b2f384073458db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.241497 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.241744 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308da528313044929b66040d6659f774. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.242250 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet a7fdbe6681894ffa8264cdfa486f16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.242242 890 tablet_service.cc:1505] Processing CreateTablet for tablet 308da528313044929b66040d6659f774 (DEFAULT_TABLE table=default.Table_6344489711f94203b5eae7cf63255c61 [id=205b9176bb44494b9b2f384073458db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.242815 897 tablet_service.cc:1505] Processing CreateTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c (DEFAULT_TABLE table=default.Table_43e66e8dbdf240a1b5eb4233ece857c7 [id=26886277edf44c6cbffbd904706dc430]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.243803 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42de32d87ebb451090e0310e36e64277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.247730 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_de39508eb70f43d880e7bb88059d8ade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.248399 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 consensus state has no leader
I20251029 08:52:11.248358 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c (DEFAULT_TABLE table=default.Table_43e66e8dbdf240a1b5eb4233ece857c7 [id=26886277edf44c6cbffbd904706dc430]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.248764 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cac145320cf48b4aaf4adb0b686ba8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.249068 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
W20251029 08:52:11.249902 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 74fe974a368a4afe8a5da6dfd1036400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fe974a368a4afe8a5da6dfd1036400 already in progress: creating tablet
W20251029 08:52:11.250229 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bd0c43367a8c47f390cc61d2764d61e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd0c43367a8c47f390cc61d2764d61e4 already in progress: creating tablet
I20251029 08:52:11.251271 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_01fe9fda4b2c432cba9c90b4f48f52aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.251799 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01fe9fda4b2c432cba9c90b4f48f52aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.252535 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308da528313044929b66040d6659f774. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.250749 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a69aba63f41c4596a1017f38f6ed6532" } modify_external_catalogs: true
I20251029 08:52:11.252985 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a69aba63f41c4596a1017f38f6ed6532" } modify_external_catalogs: true
I20251029 08:52:11.251017 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_82ec1f06f21147c991f019803452df8c" } modify_external_catalogs: true
I20251029 08:52:11.254349 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_82ec1f06f21147c991f019803452df8c" } modify_external_catalogs: true
I20251029 08:52:11.256098 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8d4ea5c42591424eb1be9e9e3981cc89" } modify_external_catalogs: true
I20251029 08:52:11.256316 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8d4ea5c42591424eb1be9e9e3981cc89" } modify_external_catalogs: true
I20251029 08:52:11.257298 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cac145320cf48b4aaf4adb0b686ba8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.239915 1178 ts_tablet_manager.cc:1434] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.101s user 0.001s sys 0.001s
W20251029 08:52:11.241281 718 leader_election.cc:343] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.258374 718 leader_election.cc:304] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:11.258822 1704 tablet.cc:1722] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:11.259331 1687 raft_consensus.cc:2749] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:11.241727 761 tablet_service.cc:1505] Processing CreateTablet for tablet 42de32d87ebb451090e0310e36e64277 (DEFAULT_TABLE table=default.Table_007a17f442c74a71a6bb3775642f35fd [id=3ca17547257d4c358a173ebef3ef4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.241968 759 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:11.242045 720 leader_election.cc:343] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:11.259975 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 98729.8us
W20251029 08:52:11.261027 861 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:11.261190 861 leader_election.cc:304] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:11.261376 861 raft_consensus.cc:2686] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:11.261921 1707 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 08:52:11.262852 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.263417 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.264779 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4c4050fe044d04b76768d39ee7db7e (DEFAULT_TABLE table=default.Table_e747c0eb07df4d70925a7232b448b39e [id=0f792c674a4d4d21a053da8e08734316]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.265189 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4c4050fe044d04b76768d39ee7db7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.249233 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de39508eb70f43d880e7bb88059d8ade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:11.271616 718 leader_election.cc:343] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.272336 718 leader_election.cc:304] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:11.272737 1687 raft_consensus.cc:2749] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:11.273253 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.273713 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eaf12e45c54bd1b0aedc647dc90436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.247506 760 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.247663 762 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.275316 759 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4c4050fe044d04b76768d39ee7db7e (DEFAULT_TABLE table=default.Table_e747c0eb07df4d70925a7232b448b39e [id=0f792c674a4d4d21a053da8e08734316]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.275775 760 tablet_service.cc:1505] Processing CreateTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c (DEFAULT_TABLE table=default.Table_43e66e8dbdf240a1b5eb4233ece857c7 [id=26886277edf44c6cbffbd904706dc430]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.276185 762 tablet_service.cc:1505] Processing CreateTablet for tablet 308da528313044929b66040d6659f774 (DEFAULT_TABLE table=default.Table_6344489711f94203b5eae7cf63255c61 [id=205b9176bb44494b9b2f384073458db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.276582 745 tablet_service.cc:1505] Processing CreateTablet for tablet 5f67e7b28b4240618e1817bb47d5633d (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.280571 1704 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:39982 (request call id 372) took 1822 ms. Trace:
I20251029 08:52:11.280745 1704 rpcz_store.cc:276] 1029 08:52:09.457749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.457820 (+ 71us) service_pool.cc:225] Handling call
1029 08:52:11.280507 (+1822687us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:09.458964 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 08:52:09.459046 (+ 82us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 08:52:09.459054 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
1029 08:52:09.459060 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 08:52:09.459125 (+ 65us) alter_schema_op.cc:128] START. Timestamp: 1761727929459089
1029 08:52:09.459128 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:09.484603 (+ 25475us) log.cc:844] Serialized 173 byte log entry
1029 08:52:11.256476 (+1771873us) op_driver.cc:464] REPLICATION: finished
1029 08:52:11.258773 (+ 2297us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 08:52:11.259197 (+ 424us) log.cc:844] Serialized 5 byte log entry
1029 08:52:11.280124 (+ 20927us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.280421 (+ 297us) log.cc:844] Serialized 14 byte log entry
1029 08:52:11.280464 (+ 43us) alter_schema_op.cc:73] Released schema lock
1029 08:52:11.280469 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2225,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":48447,"prepare.run_wall_time_us":443410,"replication_time_us":1797247,"thread_start_us":414843,"threads_started":1,"wal-append.queue_time_us":353384,"wal-append.run_cpu_time_us":562,"wal-append.run_wall_time_us":1000715}]]}
I20251029 08:52:11.281869 898 tablet_service.cc:1505] Processing CreateTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.281906 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_538d255b7c7e4f9fbe5f5ee8236746b7" } modify_external_catalogs: true
I20251029 08:52:11.282119 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_538d255b7c7e4f9fbe5f5ee8236746b7" } modify_external_catalogs: true
I20251029 08:52:11.282337 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eaf12e45c54bd1b0aedc647dc90436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.284094 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 5f67e7b28b4240618e1817bb47d5633d (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.284586 892 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4c4050fe044d04b76768d39ee7db7e (DEFAULT_TABLE table=default.Table_e747c0eb07df4d70925a7232b448b39e [id=0f792c674a4d4d21a053da8e08734316]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.284828 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f67e7b28b4240618e1817bb47d5633d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.284991 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4c4050fe044d04b76768d39ee7db7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.287513 1179 ts_tablet_manager.cc:1916] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.287703 896 tablet_service.cc:1505] Processing CreateTablet for tablet 5f67e7b28b4240618e1817bb47d5633d (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.288077 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f67e7b28b4240618e1817bb47d5633d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.292619 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2c83cedffb0e46e9be2088d13c2da515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.293345 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c83cedffb0e46e9be2088d13c2da515 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.296223 1183 tablet_bootstrap.cc:492] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:11.296476 1183 ts_tablet_manager.cc:1403] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.234s user 0.003s sys 0.002s
I20251029 08:52:11.208580 1717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:50036 (request call id 376) took 1791 ms. Trace:
I20251029 08:52:11.212868 1691 raft_consensus.cc:493] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:11.296833 1691 raft_consensus.cc:515] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.297367 1183 raft_consensus.cc:359] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.297513 1691 leader_election.cc:290] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:11.297564 1183 raft_consensus.cc:385] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.297633 1183 raft_consensus.cc:740] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.297873 1183 consensus_queue.cc:260] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.298302 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09759a50a82f4b758449da9fbbc1f72d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:11.298403 1183 ts_tablet_manager.cc:1434] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:11.298355 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_718f9986763c4d48afe14af4b45c2f0d" } new_table_name: "default.Table_468fccd2f79d4a28b65f8ce0a6ba6340" modify_external_catalogs: true
W20251029 08:52:11.298926 861 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:11.299731 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66329e1e548949cc9d59097953288375 consensus state has no leader
I20251029 08:52:11.299366 1717 rpcz_store.cc:276] 1029 08:52:09.416921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.419622 (+ 2701us) service_pool.cc:225] Handling call
1029 08:52:11.208525 (+1788903us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:09.542410 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 08:52:09.542506 (+ 96us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 08:52:10.045466 (+502960us) alter_schema_op.cc:66] Acquired schema lock
1029 08:52:10.045473 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 08:52:10.045523 (+ 50us) alter_schema_op.cc:128] START. Timestamp: 1761727930045453
1029 08:52:10.045527 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 08:52:10.046024 (+ 497us) log.cc:844] Serialized 173 byte log entry
1029 08:52:10.680735 (+634711us) op_driver.cc:464] REPLICATION: finished
1029 08:52:11.021873 (+341138us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 08:52:11.090557 (+ 68684us) log.cc:844] Serialized 5 byte log entry
1029 08:52:11.165180 (+ 74623us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.208103 (+ 42923us) log.cc:844] Serialized 14 byte log entry
1029 08:52:11.208479 (+ 376us) alter_schema_op.cc:73] Released schema lock
1029 08:52:11.208486 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":341044,"prepare.queue_time_us":122401,"prepare.run_cpu_time_us":41893,"prepare.run_wall_time_us":503685,"replication_time_us":635178,"thread_start_us":409921,"threads_started":2,"wal-append.queue_time_us":59338}]]}
I20251029 08:52:11.300101 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_77fe8da38eac41a6968a6e26fd81d040" } modify_external_catalogs: true
I20251029 08:52:11.300282 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_77fe8da38eac41a6968a6e26fd81d040" } modify_external_catalogs: true
I20251029 08:52:11.301822 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_80786b11f8074ce283d692ae22d9e92f" } new_table_name: "default.Table_79896e97f2934c50a068bf4beaf34966" modify_external_catalogs: true
I20251029 08:52:11.303107 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet be706771a95e45e5a20a15614eadfe1a
I20251029 08:52:11.303880 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 840ff254244f43a18679232279e02d1f
I20251029 08:52:11.303941 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet be706771a95e45e5a20a15614eadfe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.304234 751 tablet_service.cc:1552] Processing DeleteTablet for tablet be706771a95e45e5a20a15614eadfe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.305234 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.305408 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.305454 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e0fa089fce564af4b03aa323cec6cbf6
W20251029 08:52:11.308543 861 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:11.308490 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1c2e24ece1404009b4b7765c555b549f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.308713 861 leader_election.cc:304] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:11.309123 1691 raft_consensus.cc:2749] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:11.309366 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ad33e51ee5e24820a2044991c72db31b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.309851 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad33e51ee5e24820a2044991c72db31b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.311848 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_344943c2bd4d4ea1bf1095ed952a2dd2" } new_table_name: "default.Table_80492c8792f54daaa09fb96f4fa0b0f5" modify_external_catalogs: true
W20251029 08:52:11.313076 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2e24ece1404009b4b7765c555b549f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.318475 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42de32d87ebb451090e0310e36e64277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.323879 1178 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.324971 1177 tablet_bootstrap.cc:654] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.325582 1178 tablet_bootstrap.cc:654] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.327143 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4c4050fe044d04b76768d39ee7db7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.330660 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:11.331168 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cac145320cf48b4aaf4adb0b686ba8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.334061 1175 log.cc:1199] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:11.334803 1175 ts_tablet_manager.cc:1950] T a7fdbe6681894ffa8264cdfa486f16e6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:11.335148 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 347) took 2551 ms. Trace:
I20251029 08:52:11.335494 1175 rpcz_store.cc:276] 1029 08:52:08.784147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.784426 (+ 279us) service_pool.cc:225] Handling call
1029 08:52:11.149345 (+2364919us) ts_tablet_manager.cc:1151] Deleting tablet a7fdbe6681894ffa8264cdfa486f16e6
1029 08:52:11.149354 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.171894 (+ 22540us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.175283 (+ 3389us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.335097 (+159814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202164}
I20251029 08:52:11.335971 1175 tablet_replica.cc:333] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:11.336179 1175 raft_consensus.cc:2243] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.336406 1175 raft_consensus.cc:2272] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.336558 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308da528313044929b66040d6659f774. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.343331 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.344133 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f67e7b28b4240618e1817bb47d5633d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.345290 582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 50887us
I20251029 08:52:11.346705 889 tablet_service.cc:1552] Processing DeleteTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.348407 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.351464 1183 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:11.356161 886 tablet_service.cc:1552] Processing DeleteTablet for tablet be706771a95e45e5a20a15614eadfe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.354501 1183 tablet_bootstrap.cc:654] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.365123 1176 ts_tablet_manager.cc:1929] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.365275 1176 log.cc:1199] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:11.365646 1176 ts_tablet_manager.cc:1950] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.365657 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a7fdbe6681894ffa8264cdfa486f16e6 (table default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]) successfully deleted
I20251029 08:52:11.366401 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 363) took 2049 ms. Trace:
I20251029 08:52:11.366523 1176 rpcz_store.cc:276] 1029 08:52:09.316496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.316546 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:11.239178 (+1922632us) ts_tablet_manager.cc:1151] Deleting tablet 2e67ddf5ceb148dfad8f7be9f26cff64
1029 08:52:11.239188 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.286002 (+ 46814us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.364970 (+ 78968us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.366366 (+ 1396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786441}
I20251029 08:52:11.366801 1175 ts_tablet_manager.cc:1916] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.369216 886 tablet_service.cc:1505] Processing CreateTablet for tablet 527a83465a844849811b0f9d26428af0 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.370625 888 tablet_service.cc:1505] Processing CreateTablet for tablet d0d00a6b772d4034ac7fa764884c7081 (DEFAULT_TABLE table=default.Table_247e1528f531419d80295027ead897a0 [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.371564 896 tablet_service.cc:1505] Processing CreateTablet for tablet 85a806cc04e641d9a1c4af97a24b13de (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.372376 895 tablet_service.cc:1505] Processing CreateTablet for tablet a69beee4032f447eaa694d237cb0e4e3 (DEFAULT_TABLE table=default.Table_25ceffa901324b3e85b4d8e8cd7268a7 [id=d687628169ff44f5b2cb137a0bc8e325]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.373339 898 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9c29b93a643ec837169a20c59de33 (DEFAULT_TABLE table=default.Table_d1c5d76d3bd24eca923ed08a72bbc56b [id=af35c0b34122466c9e057ffde58e8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.373522 889 tablet_service.cc:1505] Processing CreateTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 (DEFAULT_TABLE table=default.Table_375cce62abe84482b107057a691c49f9 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.373629 1179 ts_tablet_manager.cc:1929] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:11.373708 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527a83465a844849811b0f9d26428af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.373771 1179 log.cc:1199] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aea79bd4e6844b7abac623824cabbeeb
I20251029 08:52:11.374361 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00829c7fa0fe4e379a1a3044550e5941. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.374284 897 tablet_service.cc:1505] Processing CreateTablet for tablet 2f36641b7086430a89380056e04b8aaf (DEFAULT_TABLE table=default.Table_b200db354ad94efeb37cb69294716d1d [id=bb572b68215b450c9417191114794b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.377851 597 master_service.cc:954] ReplaceTablet: received request to replace tablet a31847928db44bd4b1ebd50c5130065f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.379597 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3762da0232e4c168433f80930414895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.380185 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3762da0232e4c168433f80930414895 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.380607 1179 ts_tablet_manager.cc:1950] T aea79bd4e6844b7abac623824cabbeeb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:11.381479 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 403) took 1136 ms. Trace:
W20251029 08:52:11.381754 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cd2e9a95b6c40be9e17397a313d249b consensus state has no leader
W20251029 08:52:11.382268 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5fa181849474a559701fdb8c18d398b consensus state has no leader
I20251029 08:52:11.382758 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:11.383044 1176 tablet_replica.cc:333] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.383256 1176 raft_consensus.cc:2243] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:11.383952 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57159c264d6641afa3415d33c1d32efe consensus state has no leader
I20251029 08:52:11.384377 1176 raft_consensus.cc:2272] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:11.386894 1697 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 4834850659221715223) tablet 00000000000000000000000000000000 cur holder 0x617000022280
I20251029 08:52:11.387161 1179 rpcz_store.cc:276] 1029 08:52:10.244821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.301918 (+ 57097us) service_pool.cc:225] Handling call
1029 08:52:11.150998 (+849080us) ts_tablet_manager.cc:1151] Deleting tablet aea79bd4e6844b7abac623824cabbeeb
1029 08:52:11.151012 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.307707 (+156695us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.373462 (+ 65755us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.381443 (+ 7981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":848833}
I20251029 08:52:11.384176 1178 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.387828 1178 ts_tablet_manager.cc:1403] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.004s
I20251029 08:52:11.388803 1178 raft_consensus.cc:359] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.389071 1178 raft_consensus.cc:385] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.389186 1178 raft_consensus.cc:740] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.389514 1178 consensus_queue.cc:260] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.391731 1178 ts_tablet_manager.cc:1434] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:11.392139 1178 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.393560 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:11.393978 1178 tablet_bootstrap.cc:654] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.395258 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 416) took 1070 ms. Trace:
I20251029 08:52:11.395455 1179 rpcz_store.cc:276] 1029 08:52:10.324448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.324687 (+ 239us) service_pool.cc:225] Handling call
1029 08:52:11.395075 (+1070388us) ts_tablet_manager.cc:1151] Deleting tablet fa8884bb8e64474d936c8825d1ef3545
1029 08:52:11.395089 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.395222 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070150}
I20251029 08:52:11.396252 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 417) took 1071 ms. Trace:
I20251029 08:52:11.397776 1179 rpcz_store.cc:276] 1029 08:52:10.324545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.325008 (+ 463us) service_pool.cc:225] Handling call
1029 08:52:11.396119 (+1071111us) ts_tablet_manager.cc:1151] Deleting tablet ba1bcb3f6c134de2b322696501927a16
1029 08:52:11.396127 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.396218 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009934}
I20251029 08:52:11.398361 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 418) took 1073 ms. Trace:
I20251029 08:52:11.394680 1176 ts_tablet_manager.cc:1916] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.400267 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f21ea257e24940f8913b8c443456da25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.400902 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f21ea257e24940f8913b8c443456da25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.403170 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:11.404662 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5038a1ad6f534f59a938f5b34e5e27db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.404673 1179 rpcz_store.cc:276] 1029 08:52:10.325163 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.325528 (+ 365us) service_pool.cc:225] Handling call
1029 08:52:11.398224 (+1072696us) ts_tablet_manager.cc:1151] Deleting tablet add0605edf5c4511ae4c8d9cb7f0babb
1029 08:52:11.398234 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.398332 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009386}
I20251029 08:52:11.405365 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 415) took 1081 ms. Trace:
W20251029 08:52:11.405454 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5038a1ad6f534f59a938f5b34e5e27db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.408746 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:11.408919 1179 rpcz_store.cc:276] 1029 08:52:10.324335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.324594 (+ 259us) service_pool.cc:225] Handling call
1029 08:52:11.405143 (+1080549us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:11.405152 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.405320 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016151}
I20251029 08:52:11.409317 597 catalog_manager.cc:6719] ReplaceTablet: tablet a31847928db44bd4b1ebd50c5130065f deleted and replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd
I20251029 08:52:11.412319 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 414) took 1088 ms. Trace:
I20251029 08:52:11.412505 1179 rpcz_store.cc:276] 1029 08:52:10.324169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.328543 (+ 4374us) service_pool.cc:225] Handling call
1029 08:52:11.412154 (+1083611us) ts_tablet_manager.cc:1151] Deleting tablet 2f40aafe7dc94851a67abd84d80d7d7b
1029 08:52:11.412166 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.412283 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023052}
I20251029 08:52:11.413398 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.414892 764 tablet_service.cc:1505] Processing CreateTablet for tablet 2f36641b7086430a89380056e04b8aaf (DEFAULT_TABLE table=default.Table_b200db354ad94efeb37cb69294716d1d [id=bb572b68215b450c9417191114794b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.415130 761 tablet_service.cc:1505] Processing CreateTablet for tablet 85a806cc04e641d9a1c4af97a24b13de (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.414893 747 tablet_service.cc:1505] Processing CreateTablet for tablet 527a83465a844849811b0f9d26428af0 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.416560 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 38032950744b4426927157fad0962dfb (DEFAULT_TABLE table=default.Table_b60ce497df09480daa025d919c6509e3 [id=4c0bf44a89564ada810be44e6fe2cd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.416990 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38032950744b4426927157fad0962dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.417812 1040 tablet_service.cc:1505] Processing CreateTablet for tablet a69beee4032f447eaa694d237cb0e4e3 (DEFAULT_TABLE table=default.Table_25ceffa901324b3e85b4d8e8cd7268a7 [id=d687628169ff44f5b2cb137a0bc8e325]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.418720 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69beee4032f447eaa694d237cb0e4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.419512 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9c29b93a643ec837169a20c59de33 (DEFAULT_TABLE table=default.Table_d1c5d76d3bd24eca923ed08a72bbc56b [id=af35c0b34122466c9e057ffde58e8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.419896 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9c29b93a643ec837169a20c59de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.420327 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 2f36641b7086430a89380056e04b8aaf (DEFAULT_TABLE table=default.Table_b200db354ad94efeb37cb69294716d1d [id=bb572b68215b450c9417191114794b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.420501 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 (DEFAULT_TABLE table=default.Table_375cce62abe84482b107057a691c49f9 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.420691 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36641b7086430a89380056e04b8aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.421000 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 2fad7549758649229ad771de926aaf15 (DEFAULT_TABLE table=default.Table_b777b26318974b9f92919063554e2100 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.421331 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fad7549758649229ad771de926aaf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.424329 1032 tablet_service.cc:1505] Processing CreateTablet for tablet a2059cb9767a4fdda690d91322511cf4 (DEFAULT_TABLE table=default.Table_d45106f234434f44ade17ce063aeacbe [id=74a8033469c64712b28568166b1ccb70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.424553 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 527a83465a844849811b0f9d26428af0 (DEFAULT_TABLE table=default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.424707 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2059cb9767a4fdda690d91322511cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.424899 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527a83465a844849811b0f9d26428af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.427095 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 85a806cc04e641d9a1c4af97a24b13de (DEFAULT_TABLE table=default.Table_1f00f2933c364d019c2b2d749778ad6d [id=6dc4e58a482742c19d68262a096d26c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.427448 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a806cc04e641d9a1c4af97a24b13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.428171 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.428417 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.428634 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d0d00a6b772d4034ac7fa764884c7081 (DEFAULT_TABLE table=default.Table_247e1528f531419d80295027ead897a0 [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.428962 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d00a6b772d4034ac7fa764884c7081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.424329 1045 tablet_service.cc:1505] Processing CreateTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 (DEFAULT_TABLE table=default.Table_183dac6abe3b4b6d8ef9614f941af3a0 [id=a99e94132c4e414ab2722fcb747ad901]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.431916 892 tablet_service.cc:1505] Processing CreateTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 (DEFAULT_TABLE table=default.Table_183dac6abe3b4b6d8ef9614f941af3a0 [id=a99e94132c4e414ab2722fcb747ad901]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.432507 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f1283f5a9245a599e5576a1e84b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.420648 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:11.434058 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.434329 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.435782 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00829c7fa0fe4e379a1a3044550e5941. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.417426 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aea79bd4e6844b7abac623824cabbeeb (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]) successfully deleted
I20251029 08:52:11.440255 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36641b7086430a89380056e04b8aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.440501 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f36641b7086430a89380056e04b8aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.441444 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527a83465a844849811b0f9d26428af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.442034 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a806cc04e641d9a1c4af97a24b13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.446203 890 tablet_service.cc:1505] Processing CreateTablet for tablet 38032950744b4426927157fad0962dfb (DEFAULT_TABLE table=default.Table_b60ce497df09480daa025d919c6509e3 [id=4c0bf44a89564ada810be44e6fe2cd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.446595 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9c29b93a643ec837169a20c59de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.446743 752 tablet_service.cc:1505] Processing CreateTablet for tablet a69beee4032f447eaa694d237cb0e4e3 (DEFAULT_TABLE table=default.Table_25ceffa901324b3e85b4d8e8cd7268a7 [id=d687628169ff44f5b2cb137a0bc8e325]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.447132 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69beee4032f447eaa694d237cb0e4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.448351 751 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9c29b93a643ec837169a20c59de33 (DEFAULT_TABLE table=default.Table_d1c5d76d3bd24eca923ed08a72bbc56b [id=af35c0b34122466c9e057ffde58e8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.448729 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9c29b93a643ec837169a20c59de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.449968 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2fad7549758649229ad771de926aaf15 (DEFAULT_TABLE table=default.Table_b777b26318974b9f92919063554e2100 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.449999 756 tablet_service.cc:1505] Processing CreateTablet for tablet d0d00a6b772d4034ac7fa764884c7081 (DEFAULT_TABLE table=default.Table_247e1528f531419d80295027ead897a0 [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.450302 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d00a6b772d4034ac7fa764884c7081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.451339 755 tablet_service.cc:1505] Processing CreateTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 (DEFAULT_TABLE table=default.Table_375cce62abe84482b107057a691c49f9 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.451761 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00829c7fa0fe4e379a1a3044550e5941. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.453310 763 tablet_service.cc:1505] Processing CreateTablet for tablet 2fad7549758649229ad771de926aaf15 (DEFAULT_TABLE table=default.Table_b777b26318974b9f92919063554e2100 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.453751 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fad7549758649229ad771de926aaf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.454950 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:11.440337 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fa8884bb8e64474d936c8825d1ef3545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa8884bb8e64474d936c8825d1ef3545 already in progress: creating tablet
I20251029 08:52:11.455380 757 tablet_service.cc:1505] Processing CreateTablet for tablet a2059cb9767a4fdda690d91322511cf4 (DEFAULT_TABLE table=default.Table_d45106f234434f44ade17ce063aeacbe [id=74a8033469c64712b28568166b1ccb70]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:11.455641 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ba1bcb3f6c134de2b322696501927a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba1bcb3f6c134de2b322696501927a16 already in progress: creating tablet
I20251029 08:52:11.455821 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2059cb9767a4fdda690d91322511cf4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.455855 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet add0605edf5c4511ae4c8d9cb7f0babb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet add0605edf5c4511ae4c8d9cb7f0babb already in progress: creating tablet
W20251029 08:52:11.456107 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5ac1af8b78a4d94b0e3da14ebfffd92 already in progress: creating tablet
I20251029 08:52:11.456167 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:11.456331 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2f40aafe7dc94851a67abd84d80d7d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f40aafe7dc94851a67abd84d80d7d7b already in progress: creating tablet
W20251029 08:52:11.456552 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7b5051ff776e4975b917db0393a1f945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b5051ff776e4975b917db0393a1f945 already in progress: creating tablet
I20251029 08:52:11.456786 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.456949 893 tablet_service.cc:1505] Processing CreateTablet for tablet a2059cb9767a4fdda690d91322511cf4 (DEFAULT_TABLE table=default.Table_d45106f234434f44ade17ce063aeacbe [id=74a8033469c64712b28568166b1ccb70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.457422 1177 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:11.457590 1177 ts_tablet_manager.cc:1403] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.230s user 0.002s sys 0.002s
I20251029 08:52:11.458657 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d00a6b772d4034ac7fa764884c7081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.459257 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_31ae710df5b54ffdb4d5b6897ad25536" } new_table_name: "default.Table_7501d57fa16e41269af41cad333961d2" modify_external_catalogs: true
I20251029 08:52:11.459681 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69beee4032f447eaa694d237cb0e4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.460604 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a806cc04e641d9a1c4af97a24b13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.461949 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2059cb9767a4fdda690d91322511cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.462809 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38032950744b4426927157fad0962dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.463306 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fad7549758649229ad771de926aaf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.468112 1177 raft_consensus.cc:359] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.468343 1177 raft_consensus.cc:385] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.468434 1177 raft_consensus.cc:740] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.468667 1177 consensus_queue.cc:260] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:11.468998 861 consensus_peers.cc:597] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8cb0c6c6a1c4877a89476d4822f48db. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:11.469238 1177 ts_tablet_manager.cc:1434] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 08:52:11.469777 1177 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:11.475417 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85df43bf781c421688f163331de0eb2b consensus state has no leader
I20251029 08:52:11.482209 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
I20251029 08:52:11.484438 759 tablet_service.cc:1505] Processing CreateTablet for tablet 38032950744b4426927157fad0962dfb (DEFAULT_TABLE table=default.Table_b60ce497df09480daa025d919c6509e3 [id=4c0bf44a89564ada810be44e6fe2cd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.485370 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38032950744b4426927157fad0962dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.485999 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.488837 1177 tablet_bootstrap.cc:654] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.491477 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]). Expected version 2 got 0
I20251029 08:52:11.492884 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.493307 762 tablet_service.cc:1505] Processing CreateTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 (DEFAULT_TABLE table=default.Table_183dac6abe3b4b6d8ef9614f941af3a0 [id=a99e94132c4e414ab2722fcb747ad901]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.493831 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f1283f5a9245a599e5576a1e84b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.494652 760 tablet_service.cc:1505] Processing CreateTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.495074 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eaf12e45c54bd1b0aedc647dc90436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.495700 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.498920 896 tablet_service.cc:1505] Processing CreateTablet for tablet 8df243ce9eba41b7a050706733d7f441 (DEFAULT_TABLE table=default.Table_ad33e51ee5e24820a2044991c72db31b [id=c52711e74a90460fba6f61ae5ec27977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.499413 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df243ce9eba41b7a050706733d7f441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.500133 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.500521 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cf906c73dc84403bcc3dece74a822bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2d11b75de7d448c68fd63ab1ae81a464) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.512511 1687 raft_consensus.cc:493] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:11.513729 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4a022119f31d4460af5cdd68baa8521b" } modify_external_catalogs: true
I20251029 08:52:11.513996 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4a022119f31d4460af5cdd68baa8521b" } modify_external_catalogs: true
I20251029 08:52:11.515411 1687 raft_consensus.cc:515] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.515508 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_620a010c6d044d05af5e6a82066c21c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.516147 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_620a010c6d044d05af5e6a82066c21c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.516587 1687 leader_election.cc:290] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:11.517520 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:11.518226 718 leader_election.cc:343] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.518474 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7759e5e441aa4f8aa0f841d81c0cdd52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.519016 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7759e5e441aa4f8aa0f841d81c0cdd52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.520269 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11a78e5b37424289a184f8bccbd263d4" } modify_external_catalogs: true
I20251029 08:52:11.520437 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11a78e5b37424289a184f8bccbd263d4" } modify_external_catalogs: true
I20251029 08:52:11.521648 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ea2ca0e66f9c40d1971ec171d61737fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.522102 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea2ca0e66f9c40d1971ec171d61737fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.523450 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_468fccd2f79d4a28b65f8ce0a6ba6340" } new_table_name: "default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf" modify_external_catalogs: true
I20251029 08:52:11.524293 902 tablet_service.cc:1505] Processing CreateTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c (DEFAULT_TABLE table=default.Table_d6e32e655391437ea03d14eade0cc35f [id=ab4b38f7d715445b80c726527cc3419a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.524792 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6008b4e6243ccaf166a31f1a9f62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.525923 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:11.528203 720 leader_election.cc:343] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.528362 720 leader_election.cc:304] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:11.529353 1687 raft_consensus.cc:2749] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:11.533951 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.534627 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.539755 1175 ts_tablet_manager.cc:1929] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.539994 1175 log.cc:1199] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a2947083e3e447d5addae5effd186eb8
I20251029 08:52:11.540235 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d5bb0fd4c62d47a2b10719ebd55851c0
I20251029 08:52:11.540985 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:11.542191 890 tablet_service.cc:1505] Processing CreateTablet for tablet d2941ac88a7d453bb36f311c363231ab (DEFAULT_TABLE table=default.Table_de39508eb70f43d880e7bb88059d8ade [id=4ea290dd497d471bab20338701dfde2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.542845 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2941ac88a7d453bb36f311c363231ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.543018 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.542471 897 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.544560 1175 ts_tablet_manager.cc:1950] T a2947083e3e447d5addae5effd186eb8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:11.554128 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5fa181849474a559701fdb8c18d398b consensus state has no leader
I20251029 08:52:11.555091 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1c5d76d3bd24eca923ed08a72bbc56b" } modify_external_catalogs: true
I20251029 08:52:11.555426 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d1c5d76d3bd24eca923ed08a72bbc56b" } modify_external_catalogs: true
I20251029 08:52:11.559705 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 8df243ce9eba41b7a050706733d7f441 (DEFAULT_TABLE table=default.Table_ad33e51ee5e24820a2044991c72db31b [id=c52711e74a90460fba6f61ae5ec27977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.562844 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df243ce9eba41b7a050706733d7f441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.562963 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 426) took 1125 ms. Trace:
I20251029 08:52:11.563033 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 348) took 2764 ms. Trace:
I20251029 08:52:11.563108 1179 rpcz_store.cc:276] 1029 08:52:10.437499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.438433 (+ 934us) service_pool.cc:225] Handling call
1029 08:52:11.562781 (+1124348us) ts_tablet_manager.cc:1151] Deleting tablet 7a355c68ecfb45ff8c396873d91268a8
1029 08:52:11.562794 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.562924 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102319}
I20251029 08:52:11.563555 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 441) took 1025 ms. Trace:
I20251029 08:52:11.563719 1179 rpcz_store.cc:276] 1029 08:52:10.537874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.594560 (+ 56686us) service_pool.cc:225] Handling call
1029 08:52:11.563405 (+968845us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:11.563414 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.563514 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":958432}
I20251029 08:52:11.564808 1183 tablet_bootstrap.cc:492] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:11.564965 1183 ts_tablet_manager.cc:1403] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.214s user 0.012s sys 0.015s
I20251029 08:52:11.566820 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 49d9c29b93a643ec837169a20c59de33
I20251029 08:52:11.568336 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9c29b93a643ec837169a20c59de33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.571309 1438 consensus_queue.cc:1048] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:11.572391 1179 tablet_replica.cc:333] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:11.572819 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 652903cbbdb042d780e7d71fc2c0ab4d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.573527 760 tablet_service.cc:1505] Processing CreateTablet for tablet 8df243ce9eba41b7a050706733d7f441 (DEFAULT_TABLE table=default.Table_ad33e51ee5e24820a2044991c72db31b [id=c52711e74a90460fba6f61ae5ec27977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.574242 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df243ce9eba41b7a050706733d7f441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.580122 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f1283f5a9245a599e5576a1e84b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.581470 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 652903cbbdb042d780e7d71fc2c0ab4d
I20251029 08:52:11.581972 589 catalog_manager.cc:6719] ReplaceTablet: tablet 652903cbbdb042d780e7d71fc2c0ab4d deleted and replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4
I20251029 08:52:11.580293 1175 rpcz_store.cc:276] 1029 08:52:08.784354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.784718 (+ 364us) service_pool.cc:225] Handling call
1029 08:52:11.335889 (+2551171us) ts_tablet_manager.cc:1151] Deleting tablet a2947083e3e447d5addae5effd186eb8
1029 08:52:11.335901 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.481207 (+145306us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.538772 (+ 57565us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.549260 (+ 10488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386153}
I20251029 08:52:11.582616 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 350) took 2612 ms. Trace:
I20251029 08:52:11.583316 1175 rpcz_store.cc:276] 1029 08:52:08.970063 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.970313 (+ 250us) service_pool.cc:225] Handling call
1029 08:52:11.582442 (+2612129us) ts_tablet_manager.cc:1151] Deleting tablet de78ea608c904aeea7f7145b9c80f45b
1029 08:52:11.582453 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.582576 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2611880}
I20251029 08:52:11.584492 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a2947083e3e447d5addae5effd186eb8 (table default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]) successfully deleted
I20251029 08:52:11.585265 756 tablet_service.cc:1505] Processing CreateTablet for tablet 47470a907c1949989f79e8a2b803dc25 (DEFAULT_TABLE table=default.Table_01fe9fda4b2c432cba9c90b4f48f52aa [id=bc8a939fa4ab47049d9b2b82b4acdb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.585322 759 tablet_service.cc:1505] Processing CreateTablet for tablet d2941ac88a7d453bb36f311c363231ab (DEFAULT_TABLE table=default.Table_de39508eb70f43d880e7bb88059d8ade [id=4ea290dd497d471bab20338701dfde2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.585708 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47470a907c1949989f79e8a2b803dc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.587209 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7c15ab0db646407ab8460031c72ab852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.587690 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fb40280e4c7d421889ad47f81c4574b2" } new_table_name: "default.Table_4b0593e0dafd4dfda7b1b2b9fb545383" modify_external_catalogs: true
W20251029 08:52:11.587867 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c15ab0db646407ab8460031c72ab852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.589061 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8f9777707145473e941f784ae1963744" } new_table_name: "default.Table_a565b344ef8e4b60b07cb259f877df2a" modify_external_catalogs: true
I20251029 08:52:11.590401 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2eb98ef2bcb744d9b4560b816d57442d" } modify_external_catalogs: true
I20251029 08:52:11.590574 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2eb98ef2bcb744d9b4560b816d57442d" } modify_external_catalogs: true
I20251029 08:52:11.590637 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7f838f98d4a34099a018852bfa404b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.591120 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f838f98d4a34099a018852bfa404b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.590417 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_897033c1be1f463fb7ef32a818c87de3" } new_table_name: "default.Table_f096ddf6ddeb4b64b780d57471d56d8d" modify_external_catalogs: true
I20251029 08:52:11.593284 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_82984426f7eb4edf9c0cab2e8544d1dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.593755 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82984426f7eb4edf9c0cab2e8544d1dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:11.597396 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet de78ea608c904aeea7f7145b9c80f45b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de78ea608c904aeea7f7145b9c80f45b already in progress: creating tablet
I20251029 08:52:11.598043 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 351) took 2627 ms. Trace:
I20251029 08:52:11.598172 1175 rpcz_store.cc:276] 1029 08:52:08.970230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.970599 (+ 369us) service_pool.cc:225] Handling call
1029 08:52:11.597865 (+2627266us) ts_tablet_manager.cc:1151] Deleting tablet a3ba512bd4f04bd9aceb4cb210fe9689
1029 08:52:11.597880 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.597998 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627108}
I20251029 08:52:11.598866 1178 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.599074 1178 ts_tablet_manager.cc:1403] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.207s user 0.004s sys 0.000s
I20251029 08:52:11.599977 1178 raft_consensus.cc:359] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.600260 1178 raft_consensus.cc:385] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.600404 1178 raft_consensus.cc:740] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.600768 1178 consensus_queue.cc:260] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:11.600903 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c32679cbb1884f09addb0f7c26ac941e consensus state has no leader
I20251029 08:52:11.605721 764 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.606024 1178 ts_tablet_manager.cc:1434] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20251029 08:52:11.608846 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7a355c68ecfb45ff8c396873d91268a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a355c68ecfb45ff8c396873d91268a8 already in progress: creating tablet
W20251029 08:52:11.609158 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 657cdaad97e24611a997e26a6293ea5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657cdaad97e24611a997e26a6293ea5e already in progress: creating tablet
W20251029 08:52:11.609510 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet aa411391c63146e7acb8b4c6edbe12a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa411391c63146e7acb8b4c6edbe12a0 already in progress: creating tablet
I20251029 08:52:11.609730 888 tablet_service.cc:1505] Processing CreateTablet for tablet 6a8c25ba4e9c410fa6d12949d6e595d7 (DEFAULT_TABLE table=default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a [id=1e36c914f03b48eaa7061b7731ba512a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:11.609814 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b7da7e08b74f4a9aacedada42e839a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7da7e08b74f4a9aacedada42e839a11 already in progress: creating tablet
W20251029 08:52:11.609843 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a3ba512bd4f04bd9aceb4cb210fe9689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ba512bd4f04bd9aceb4cb210fe9689 already in progress: creating tablet
W20251029 08:52:11.610033 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet aa4f97075b3941aaa7cda3f9413e867f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa4f97075b3941aaa7cda3f9413e867f already in progress: creating tablet
I20251029 08:52:11.610205 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a8c25ba4e9c410fa6d12949d6e595d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.610257 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet aafa846cb2144cec9ce4b28e61d91961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafa846cb2144cec9ce4b28e61d91961 already in progress: creating tablet
I20251029 08:52:11.610348 897 tablet_service.cc:1505] Processing CreateTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:11.610483 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 39fc9612689b46d498d6092bd552da4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39fc9612689b46d498d6092bd552da4e already in progress: creating tablet
I20251029 08:52:11.610833 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017e8fe5f634ef0bf7f8be7c8aea033. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.612154 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e3c24c0d32f74435bd9d28f2415e2c47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c24c0d32f74435bd9d28f2415e2c47 already in progress: creating tablet
W20251029 08:52:11.612370 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 43c3074caee74069a8a7f1b87721d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43c3074caee74069a8a7f1b87721d726 already in progress: creating tablet
W20251029 08:52:11.612524 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9907d044daa145168417371c2bd66544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9907d044daa145168417371c2bd66544 already in progress: creating tablet
I20251029 08:52:11.612524 1687 raft_consensus.cc:493] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:11.612687 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet af38662420d14ac1a7cb5fd03fc1b9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af38662420d14ac1a7cb5fd03fc1b9d8 already in progress: creating tablet
I20251029 08:52:11.612691 1687 raft_consensus.cc:515] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:11.612852 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d50e2fdfc0e64516a41c90f5a56f4c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d50e2fdfc0e64516a41c90f5a56f4c89 already in progress: creating tablet
W20251029 08:52:11.613018 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e8c4d793f10147fba5e4f44474575bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8c4d793f10147fba5e4f44474575bf0 already in progress: creating tablet
I20251029 08:52:11.613363 1687 leader_election.cc:290] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:11.614794 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.616449 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add0605edf5c4511ae4c8d9cb7f0babb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:11.617429 718 leader_election.cc:343] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.624307 1178 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.626695 1178 tablet_bootstrap.cc:654] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.629649 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2941ac88a7d453bb36f311c363231ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.613173 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 351047c725f8488abbd074bd40790b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351047c725f8488abbd074bd40790b48 already in progress: creating tablet
W20251029 08:52:11.615170 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6689cf0804f946c18ebc555a191549d4 consensus state has no leader
W20251029 08:52:11.634867 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e4cdf03acb92449d847aec897868b967 because the tablet was not found. No further retry: Not found: Tablet not found: e4cdf03acb92449d847aec897868b967
I20251029 08:52:11.634130 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fc047f205272431fa1ba26215c0013d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.636751 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc047f205272431fa1ba26215c0013d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.636864 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c" } modify_external_catalogs: true
I20251029 08:52:11.637073 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c" } modify_external_catalogs: true
I20251029 08:52:11.633721 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e65b3b1c8d14743aca1465d95ace95f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.597429 1043 tablet_service.cc:1505] Processing CreateTablet for tablet d2941ac88a7d453bb36f311c363231ab (DEFAULT_TABLE table=default.Table_de39508eb70f43d880e7bb88059d8ade [id=4ea290dd497d471bab20338701dfde2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.639465 1179 raft_consensus.cc:2243] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.639657 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2941ac88a7d453bb36f311c363231ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.639714 1179 raft_consensus.cc:2272] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.640461 1179 ts_tablet_manager.cc:1916] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.597858 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 47470a907c1949989f79e8a2b803dc25 (DEFAULT_TABLE table=default.Table_01fe9fda4b2c432cba9c90b4f48f52aa [id=bc8a939fa4ab47049d9b2b82b4acdb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.604759 1044 tablet_service.cc:1505] Processing CreateTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c (DEFAULT_TABLE table=default.Table_d6e32e655391437ea03d14eade0cc35f [id=ab4b38f7d715445b80c726527cc3419a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.646803 1176 ts_tablet_manager.cc:1929] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.605237 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.606462 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.646986 1176 log.cc:1199] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:11.647339 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47470a907c1949989f79e8a2b803dc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.639117 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9c29b93a643ec837169a20c59de33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:11.648974 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cc95257c937454aa6cd55b5a9394a56 consensus state has no leader
I20251029 08:52:11.649441 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.650105 1176 ts_tablet_manager.cc:1950] T 9165c4bf484a48b4a0b7246cb11571bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.650247 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.651048 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 364) took 2333 ms. Trace:
I20251029 08:52:11.651324 756 tablet_service.cc:1505] Processing CreateTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c (DEFAULT_TABLE table=default.Table_d6e32e655391437ea03d14eade0cc35f [id=ab4b38f7d715445b80c726527cc3419a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.651206 1176 rpcz_store.cc:276] 1029 08:52:09.317491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.317549 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:11.382978 (+2065429us) ts_tablet_manager.cc:1151] Deleting tablet 9165c4bf484a48b4a0b7246cb11571bf
1029 08:52:11.382986 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.573047 (+190061us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.646652 (+ 73605us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.650977 (+ 4325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1930077}
I20251029 08:52:11.651784 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6008b4e6243ccaf166a31f1a9f62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.652034 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 366) took 2199 ms. Trace:
I20251029 08:52:11.652110 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6008b4e6243ccaf166a31f1a9f62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.652143 1176 rpcz_store.cc:276] 1029 08:52:09.452438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.452892 (+ 454us) service_pool.cc:225] Handling call
1029 08:52:11.651846 (+2198954us) ts_tablet_manager.cc:1151] Deleting tablet ea13e9ea2c11479ea86e3fd7f2a9e59c
1029 08:52:11.651855 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.651956 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197508}
I20251029 08:52:11.652480 1176 tablet_replica.cc:333] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.652690 1176 raft_consensus.cc:2243] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.652923 1176 raft_consensus.cc:2272] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.653178 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 352) took 2670 ms. Trace:
I20251029 08:52:11.653321 1175 rpcz_store.cc:276] 1029 08:52:08.982240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:08.982308 (+ 68us) service_pool.cc:225] Handling call
1029 08:52:11.653017 (+2670709us) ts_tablet_manager.cc:1151] Deleting tablet 4e2773cae9814df79ee60ea8b0fe4e9d
1029 08:52:11.653029 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.653136 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670473}
W20251029 08:52:11.653786 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4e2773cae9814df79ee60ea8b0fe4e9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2773cae9814df79ee60ea8b0fe4e9d already in progress: creating tablet
I20251029 08:52:11.654072 1176 ts_tablet_manager.cc:1916] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.656776 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 6a8c25ba4e9c410fa6d12949d6e595d7 (DEFAULT_TABLE table=default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a [id=1e36c914f03b48eaa7061b7731ba512a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.584374 896 tablet_service.cc:1505] Processing CreateTablet for tablet 47470a907c1949989f79e8a2b803dc25 (DEFAULT_TABLE table=default.Table_01fe9fda4b2c432cba9c90b4f48f52aa [id=bc8a939fa4ab47049d9b2b82b4acdb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.596309 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.657295 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a8c25ba4e9c410fa6d12949d6e595d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.659288 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fc04616301354cfe873c825a9f5d7ab6
I20251029 08:52:11.660379 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_87bd0ffa8ae04a5da1b10cae0b440517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.660475 902 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.611095 1183 raft_consensus.cc:359] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.613766 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add0605edf5c4511ae4c8d9cb7f0babb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:11.660910 1183 raft_consensus.cc:385] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.660971 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47470a907c1949989f79e8a2b803dc25. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.660984 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bd0ffa8ae04a5da1b10cae0b440517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.661031 1183 raft_consensus.cc:740] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:11.661399 720 leader_election.cc:343] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:11.661406 1183 consensus_queue.cc:260] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.661535 720 leader_election.cc:304] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:11.636159 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9c29b93a643ec837169a20c59de33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.662184 1687 raft_consensus.cc:2749] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:11.663031 764 tablet_service.cc:1505] Processing CreateTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.663492 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017e8fe5f634ef0bf7f8be7c8aea033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.664373 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:11.664757 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 360) took 2432 ms. Trace:
I20251029 08:52:11.664872 1175 rpcz_store.cc:276] 1029 08:52:09.232089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.232271 (+ 182us) service_pool.cc:225] Handling call
1029 08:52:11.664584 (+2432313us) ts_tablet_manager.cc:1151] Deleting tablet 2723cdd513ee4e8ca6773b38f785b09e
1029 08:52:11.664596 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.664717 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2431242}
W20251029 08:52:11.665335 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2723cdd513ee4e8ca6773b38f785b09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2723cdd513ee4e8ca6773b38f785b09e already in progress: creating tablet
I20251029 08:52:11.671440 1175 tablet_replica.cc:333] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:11.671757 1175 raft_consensus.cc:2243] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:11.672152 1175 raft_consensus.cc:2272] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.672590 762 tablet_service.cc:1505] Processing CreateTablet for tablet 6a8c25ba4e9c410fa6d12949d6e595d7 (DEFAULT_TABLE table=default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a [id=1e36c914f03b48eaa7061b7731ba512a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.672823 1175 ts_tablet_manager.cc:1916] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.673071 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a8c25ba4e9c410fa6d12949d6e595d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.672704 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9e95376692ff46ceb83308b096d8cd24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.675879 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e95376692ff46ceb83308b096d8cd24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.676488 1183 ts_tablet_manager.cc:1434] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.111s user 0.001s sys 0.001s
I20251029 08:52:11.677175 902 tablet_service.cc:1505] Processing CreateTablet for tablet 633ace1413464e939e4e47782370e281 (DEFAULT_TABLE table=default.Table_ee85eba8daea413bb0586c2a113351a1 [id=a83722707d254d7f98a23c318486c019]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.677626 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ace1413464e939e4e47782370e281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.680347 888 tablet_service.cc:1505] Processing CreateTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee (DEFAULT_TABLE table=default.Table_e857915b93d94731a6672caf050b7807 [id=c0f3daf63db74c0e8d27140fd8d40c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.678017 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b200db354ad94efeb37cb69294716d1d" } new_table_name: "default.Table_4cee30efd10f47c88ab3c9c5a2df5348" modify_external_catalogs: true
I20251029 08:52:11.680725 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a712185fe4aa4f8faf1ce2df229c11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.684358 893 tablet_service.cc:1505] Processing CreateTablet for tablet 87c98206d7fa409b8017abc72688db93 (DEFAULT_TABLE table=default.Table_1c2e24ece1404009b4b7765c555b549f [id=5f1e417568044b43b5b50db23a0ecc2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.684783 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c98206d7fa409b8017abc72688db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.685114 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9165c4bf484a48b4a0b7246cb11571bf (table default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]) successfully deleted
W20251029 08:52:11.685389 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea13e9ea2c11479ea86e3fd7f2a9e59c already in progress: creating tablet
I20251029 08:52:11.686226 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.686259 761 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.687410 761 tablet_service.cc:1505] Processing CreateTablet for tablet 633ace1413464e939e4e47782370e281 (DEFAULT_TABLE table=default.Table_ee85eba8daea413bb0586c2a113351a1 [id=a83722707d254d7f98a23c318486c019]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.687637 764 tablet_service.cc:1505] Processing CreateTablet for tablet 9a678d92f6334b66989eb3f71d830257 (DEFAULT_TABLE table=default.Table_6eaf722368b14c059665538870edd00f [id=feb4a9766c4e455e8c4e2e3e55b3544e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.687839 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ace1413464e939e4e47782370e281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.688241 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a678d92f6334b66989eb3f71d830257. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.688427 751 tablet_service.cc:1505] Processing CreateTablet for tablet 85ee033a724942b0a90f12f15cba2701 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.688786 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ee033a724942b0a90f12f15cba2701. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.692327 747 tablet_service.cc:1505] Processing CreateTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee (DEFAULT_TABLE table=default.Table_e857915b93d94731a6672caf050b7807 [id=c0f3daf63db74c0e8d27140fd8d40c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.692735 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a712185fe4aa4f8faf1ce2df229c11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.693200 1177 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:11.694058 1177 ts_tablet_manager.cc:1403] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.224s user 0.005s sys 0.005s
I20251029 08:52:11.694939 1177 raft_consensus.cc:359] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.695151 1177 raft_consensus.cc:385] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.695246 1177 raft_consensus.cc:740] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.695483 1177 consensus_queue.cc:260] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.696184 1177 ts_tablet_manager.cc:1434] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:11.696343 763 tablet_service.cc:1505] Processing CreateTablet for tablet 87c98206d7fa409b8017abc72688db93 (DEFAULT_TABLE table=default.Table_1c2e24ece1404009b4b7765c555b549f [id=5f1e417568044b43b5b50db23a0ecc2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.696642 897 tablet_service.cc:1505] Processing CreateTablet for tablet 85ee033a724942b0a90f12f15cba2701 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.696740 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c98206d7fa409b8017abc72688db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.697248 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ee033a724942b0a90f12f15cba2701. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.716051 1176 ts_tablet_manager.cc:1929] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:11.716240 1176 log.cc:1199] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4b55ae84c8d441459a5931abda471f6c
I20251029 08:52:11.716815 1176 ts_tablet_manager.cc:1950] T 4b55ae84c8d441459a5931abda471f6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.716884 1178 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.717058 1178 ts_tablet_manager.cc:1403] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20251029 08:52:11.717713 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 367) took 2265 ms. Trace:
I20251029 08:52:11.717913 1178 raft_consensus.cc:359] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.718130 1178 raft_consensus.cc:385] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.718201 1178 raft_consensus.cc:740] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.718434 1178 consensus_queue.cc:260] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.718930 1178 ts_tablet_manager.cc:1434] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:11.719600 1178 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.721504 1178 tablet_bootstrap.cc:654] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.724967 889 tablet_service.cc:1505] Processing CreateTablet for tablet 9a678d92f6334b66989eb3f71d830257 (DEFAULT_TABLE table=default.Table_6eaf722368b14c059665538870edd00f [id=feb4a9766c4e455e8c4e2e3e55b3544e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.725508 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a678d92f6334b66989eb3f71d830257. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.717849 1176 rpcz_store.cc:276] 1029 08:52:09.452549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.452844 (+ 295us) service_pool.cc:225] Handling call
1029 08:52:11.652392 (+2199548us) ts_tablet_manager.cc:1151] Deleting tablet 4b55ae84c8d441459a5931abda471f6c
1029 08:52:11.652403 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.704642 (+ 52239us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.715871 (+ 11229us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.717664 (+ 1793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2194610}
I20251029 08:52:11.728520 1176 tablet_replica.cc:333] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.730708 1176 raft_consensus.cc:2243] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.732673 1177 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:11.734238 1176 raft_consensus.cc:2272] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.734287 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]) successfully deleted
I20251029 08:52:11.735265 1176 ts_tablet_manager.cc:1916] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.739823 1177 tablet_bootstrap.cc:654] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.740886 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.748651 597 master_service.cc:954] ReplaceTablet: received request to replace tablet c9bbf6401423411089c9070f3b03f1a7 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:11.749501 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ee85eba8daea413bb0586c2a113351a1" } modify_external_catalogs: true
I20251029 08:52:11.749722 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ee85eba8daea413bb0586c2a113351a1" } modify_external_catalogs: true
I20251029 08:52:11.751004 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7759e5e441aa4f8aa0f841d81c0cdd52" } modify_external_catalogs: true
I20251029 08:52:11.751184 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7759e5e441aa4f8aa0f841d81c0cdd52" } modify_external_catalogs: true
I20251029 08:52:11.751652 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dd3a01ef1bab4ac1b129065e647ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.752274 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd3a01ef1bab4ac1b129065e647ffb2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.757532 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 87c98206d7fa409b8017abc72688db93 (DEFAULT_TABLE table=default.Table_1c2e24ece1404009b4b7765c555b549f [id=5f1e417568044b43b5b50db23a0ecc2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.758236 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.737126 764 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.750797 1178 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.759598 1178 ts_tablet_manager.cc:1403] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20251029 08:52:11.760666 1178 raft_consensus.cc:359] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.760859 1178 raft_consensus.cc:385] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.760958 1178 raft_consensus.cc:740] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.761178 1178 consensus_queue.cc:260] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.761778 1178 ts_tablet_manager.cc:1434] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:11.761834 1183 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:11.762130 1178 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.764525 1178 tablet_bootstrap.cc:654] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.767208 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f5b52314139945dd9c2e6ec899bf06dc
I20251029 08:52:11.770574 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:11.771263 595 catalog_manager.cc:6719] ReplaceTablet: tablet 3e65b3b1c8d14743aca1465d95ace95f deleted and replaced by tablet a186df0db24c4d22abee7ec57bcb199b
I20251029 08:52:11.771735 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5104aa104b704bfea93e338b7efc8906"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.774912 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_21761ed98ca84ce58dcd72a3dac157c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:11.775686 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21761ed98ca84ce58dcd72a3dac157c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.777559 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.777635 1178 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.777803 1178 ts_tablet_manager.cc:1403] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20251029 08:52:11.778656 1178 raft_consensus.cc:359] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.778858 1178 raft_consensus.cc:385] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.778921 1178 raft_consensus.cc:740] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.779198 1178 consensus_queue.cc:260] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.779819 1178 ts_tablet_manager.cc:1434] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251029 08:52:11.782996 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5104aa104b704bfea93e338b7efc8906 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.784302 902 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.793449 1178 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:11.795461 1178 tablet_bootstrap.cc:654] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.803068 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_81ab02d14c514f1f9c7622f3776b4552" } new_table_name: "default.Table_3eda8a3eed924142880c9b465eb759b9" modify_external_catalogs: true
I20251029 08:52:11.804275 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:11.805377 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6017008571b64d0cb74edcd9f66e78a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.805657 1177 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:11.805791 1177 ts_tablet_manager.cc:1403] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20251029 08:52:11.806757 1177 raft_consensus.cc:359] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.806924 1177 raft_consensus.cc:385] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.807281 593 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 926980f1ea64498e99fd82038b0e6926
I20251029 08:52:11.807325 1177 raft_consensus.cc:740] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:11.807639 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f1be1b978b7433896bb423e059679b9" } modify_external_catalogs: true
I20251029 08:52:11.807626 1177 consensus_queue.cc:260] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:11.807788 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f1be1b978b7433896bb423e059679b9" } modify_external_catalogs: true
W20251029 08:52:11.810828 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f36641b7086430a89380056e04b8aaf consensus state has no leader
I20251029 08:52:11.812744 1177 ts_tablet_manager.cc:1434] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:11.813095 1177 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:11.814874 1177 tablet_bootstrap.cc:654] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.815758 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 633ace1413464e939e4e47782370e281
I20251029 08:52:11.816694 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ace1413464e939e4e47782370e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.819190 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.831621 597 catalog_manager.cc:6719] ReplaceTablet: tablet c9bbf6401423411089c9070f3b03f1a7 deleted and replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e
W20251029 08:52:11.832042 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6017008571b64d0cb74edcd9f66e78a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.832479 751 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.834029 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ace1413464e939e4e47782370e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.835282 1176 ts_tablet_manager.cc:1929] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.835574 1176 log.cc:1199] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:11.836788 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_25ceffa901324b3e85b4d8e8cd7268a7" } new_table_name: "default.Table_736d75bfbe144b3887b5d49fa7088845" modify_external_catalogs: true
I20251029 08:52:11.837303 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f21ea257e24940f8913b8c443456da25" } new_table_name: "default.Table_752b2204770149368a72732e2da51c1a" modify_external_catalogs: true
I20251029 08:52:11.835973 1176 ts_tablet_manager.cc:1950] T 7cbce5d595364b4bad36d59fb532c237 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.762866 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 85ee033a724942b0a90f12f15cba2701 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.762902 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 9a678d92f6334b66989eb3f71d830257 (DEFAULT_TABLE table=default.Table_6eaf722368b14c059665538870edd00f [id=feb4a9766c4e455e8c4e2e3e55b3544e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.763049 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 633ace1413464e939e4e47782370e281 (DEFAULT_TABLE table=default.Table_ee85eba8daea413bb0586c2a113351a1 [id=a83722707d254d7f98a23c318486c019]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.831779 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 (DEFAULT_TABLE table=default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.837971 1734 raft_consensus.cc:493] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:11.838291 1045 tablet_service.cc:1505] Processing CreateTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee (DEFAULT_TABLE table=default.Table_e857915b93d94731a6672caf050b7807 [id=c0f3daf63db74c0e8d27140fd8d40c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.839090 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.839229 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.839452 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f with delete_type TABLET_DATA_DELETED (replaced by tablet c9a1d5e1f68e4f0e8b78641bbdc98123) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.839604 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ace1413464e939e4e47782370e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.839761 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.846486 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 371) took 2390 ms. Trace:
I20251029 08:52:11.846938 1176 rpcz_store.cc:276] 1029 08:52:09.455963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.456045 (+ 82us) service_pool.cc:225] Handling call
1029 08:52:11.728445 (+2272400us) ts_tablet_manager.cc:1151] Deleting tablet 7cbce5d595364b4bad36d59fb532c237
1029 08:52:11.728454 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.825794 (+ 97340us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.835144 (+ 9350us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.846405 (+ 11261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270095}
I20251029 08:52:11.847467 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7cbce5d595364b4bad36d59fb532c237 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
I20251029 08:52:11.850915 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b1aaf233eba2461f9182e094a0115a4e
I20251029 08:52:11.854322 1176 tablet_replica.cc:333] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.854843 1176 raft_consensus.cc:2243] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:11.856745 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d26ff78aa5f94ae49a197daf2be00aa5 consensus state has no leader
I20251029 08:52:11.857087 1176 raft_consensus.cc:2272] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.857760 1176 ts_tablet_manager.cc:1916] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.860903 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8c0fd828d9d14377aacc896765c045e3" } new_table_name: "default.Table_3e111656804b44da9e8317170d1f3605" modify_external_catalogs: true
I20251029 08:52:11.861608 1179 ts_tablet_manager.cc:1929] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.861150 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_051b538b21fe4b6b8ea44de581730151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.862195 1179 log.cc:1199] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a20b64ff177241309e5d59e84f0bafa2
I20251029 08:52:11.862623 1179 ts_tablet_manager.cc:1950] T a20b64ff177241309e5d59e84f0bafa2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:11.862632 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.862979 1734 raft_consensus.cc:515] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.863739 1734 leader_election.cc:290] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:11.864863 1176 ts_tablet_manager.cc:1929] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.864886 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet b1aaf233eba2461f9182e094a0115a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.864995 1176 log.cc:1199] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/28face87091b49d2971217618c95d2f5
I20251029 08:52:11.865161 1183 tablet_bootstrap.cc:654] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.865409 1176 ts_tablet_manager.cc:1950] T 28face87091b49d2971217618c95d2f5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:11.865597 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_051b538b21fe4b6b8ea44de581730151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.866292 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 365) took 2414 ms. Trace:
I20251029 08:52:11.866894 751 tablet_service.cc:1552] Processing DeleteTablet for tablet b1aaf233eba2461f9182e094a0115a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.867781 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.868091 761 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:11.868346 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.868788 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c98206d7fa409b8017abc72688db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.871316 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5fefde56f4da40239f0ccb837befc3db" } modify_external_catalogs: true
I20251029 08:52:11.871507 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5fefde56f4da40239f0ccb837befc3db" } modify_external_catalogs: true
I20251029 08:52:11.861369 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c68edbe3428b41579ae051abdb61b40d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:11.872900 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ee033a724942b0a90f12f15cba2701. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.866425 1176 rpcz_store.cc:276] 1029 08:52:09.452319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.452740 (+ 421us) service_pool.cc:225] Handling call
1029 08:52:11.847799 (+2395059us) ts_tablet_manager.cc:1151] Deleting tablet 28face87091b49d2971217618c95d2f5
1029 08:52:11.847810 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.861391 (+ 13581us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.864718 (+ 3327us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.866238 (+ 1520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387936}
I20251029 08:52:11.873800 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 28face87091b49d2971217618c95d2f5 (table default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]) successfully deleted
I20251029 08:52:11.874228 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 377) took 2390 ms. Trace:
I20251029 08:52:11.874356 1176 rpcz_store.cc:276] 1029 08:52:09.483858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.483916 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:11.874065 (+2390149us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:11.874077 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.874188 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387557}
W20251029 08:52:11.874835 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c66cbb495f5142d79a674f7d5adc505e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c66cbb495f5142d79a674f7d5adc505e already in progress: creating tablet
I20251029 08:52:11.875228 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 385) took 1925 ms. Trace:
I20251029 08:52:11.875349 1176 rpcz_store.cc:276] 1029 08:52:09.950239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.950384 (+ 145us) service_pool.cc:225] Handling call
1029 08:52:11.875064 (+1924680us) ts_tablet_manager.cc:1151] Deleting tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
1029 08:52:11.875075 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.875198 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924462}
W20251029 08:52:11.875720 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa0ff14e19b4ad7a8b33796481f0f3f already in progress: creating tablet
I20251029 08:52:11.876363 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a678d92f6334b66989eb3f71d830257. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.877390 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a69beee4032f447eaa694d237cb0e4e3 consensus state has no leader
I20251029 08:52:11.877636 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 386) took 1926 ms. Trace:
I20251029 08:52:11.877738 1176 rpcz_store.cc:276] 1029 08:52:09.951224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.951340 (+ 116us) service_pool.cc:225] Handling call
1029 08:52:11.877508 (+1926168us) ts_tablet_manager.cc:1151] Deleting tablet 5385fef404a4463595074f0c8ff822d7
1029 08:52:11.877517 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.877604 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1925977}
W20251029 08:52:11.878144 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5385fef404a4463595074f0c8ff822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5385fef404a4463595074f0c8ff822d7 already in progress: creating tablet
I20251029 08:52:11.878580 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 387) took 1926 ms. Trace:
I20251029 08:52:11.878688 1176 rpcz_store.cc:276] 1029 08:52:09.951820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.952050 (+ 230us) service_pool.cc:225] Handling call
1029 08:52:11.878436 (+1926386us) ts_tablet_manager.cc:1151] Deleting tablet 2fb380018f46465e82ac6223ecf8dd0d
1029 08:52:11.878446 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.878546 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926218}
W20251029 08:52:11.879055 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2fb380018f46465e82ac6223ecf8dd0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb380018f46465e82ac6223ecf8dd0d already in progress: creating tablet
I20251029 08:52:11.879398 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 388) took 1927 ms. Trace:
I20251029 08:52:11.879513 1176 rpcz_store.cc:276] 1029 08:52:09.952395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.952574 (+ 179us) service_pool.cc:225] Handling call
1029 08:52:11.879273 (+1926699us) ts_tablet_manager.cc:1151] Deleting tablet bc96ef1a94844a32923087c44a5bcfc8
1029 08:52:11.879282 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.879368 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926503}
W20251029 08:52:11.879752 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57804eef0fee4b2db02b1acb75fc813d has no consensus state
W20251029 08:52:11.879957 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bc96ef1a94844a32923087c44a5bcfc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc96ef1a94844a32923087c44a5bcfc8 already in progress: creating tablet
I20251029 08:52:11.880247 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 633ace1413464e939e4e47782370e281. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:11.881033 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdcc9a441c6a4d82a58262f4fdb3f00d consensus state has no leader
I20251029 08:52:11.881402 1183 tablet_bootstrap.cc:492] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:11.882273 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017e8fe5f634ef0bf7f8be7c8aea033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.887279 1183 ts_tablet_manager.cc:1403] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.197s user 0.003s sys 0.007s
I20251029 08:52:11.888108 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a712185fe4aa4f8faf1ce2df229c11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.888197 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 396) took 1881 ms. Trace:
I20251029 08:52:11.888227 1183 raft_consensus.cc:359] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.888438 1183 raft_consensus.cc:385] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:11.888339 1176 rpcz_store.cc:276] 1029 08:52:10.006939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.046011 (+ 39072us) service_pool.cc:225] Handling call
1029 08:52:11.887982 (+1841971us) ts_tablet_manager.cc:1151] Deleting tablet 8991ea78321545be85d5c480841e4751
1029 08:52:11.887997 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.888155 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841784}
I20251029 08:52:11.888525 1183 raft_consensus.cc:740] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:11.888826 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8991ea78321545be85d5c480841e4751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8991ea78321545be85d5c480841e4751 already in progress: creating tablet
I20251029 08:52:11.888795 1183 consensus_queue.cc:260] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.888927 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 420) took 1850 ms. Trace:
I20251029 08:52:11.889025 1176 rpcz_store.cc:276] 1029 08:52:10.038787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.171167 (+132380us) service_pool.cc:225] Handling call
1029 08:52:11.888793 (+1717626us) ts_tablet_manager.cc:1151] Deleting tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
1029 08:52:11.888804 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.888897 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717168}
I20251029 08:52:11.889336 1183 ts_tablet_manager.cc:1434] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251029 08:52:11.889392 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 already in progress: creating tablet
I20251029 08:52:11.889515 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 421) took 1802 ms. Trace:
I20251029 08:52:11.889611 1176 rpcz_store.cc:276] 1029 08:52:10.086989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.171645 (+ 84656us) service_pool.cc:225] Handling call
1029 08:52:11.889370 (+1717725us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:11.889381 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.889486 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709208}
W20251029 08:52:11.889961 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet eed66c3fa3f848df905a2a31fd553474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eed66c3fa3f848df905a2a31fd553474 already in progress: creating tablet
I20251029 08:52:11.890089 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 422) took 1764 ms. Trace:
I20251029 08:52:11.890188 1176 rpcz_store.cc:276] 1029 08:52:10.125774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.180192 (+ 54418us) service_pool.cc:225] Handling call
1029 08:52:11.889967 (+1709775us) ts_tablet_manager.cc:1151] Deleting tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
1029 08:52:11.889978 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.890059 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598393}
W20251029 08:52:11.890573 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d already in progress: creating tablet
I20251029 08:52:11.890661 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 423) took 1764 ms. Trace:
I20251029 08:52:11.890754 1176 rpcz_store.cc:276] 1029 08:52:10.125928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.291618 (+165690us) service_pool.cc:225] Handling call
1029 08:52:11.890532 (+1598914us) ts_tablet_manager.cc:1151] Deleting tablet bf29e030892d44e1ad4d1ca4f5aeab3b
1029 08:52:11.890542 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.890631 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587434}
W20251029 08:52:11.891077 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bf29e030892d44e1ad4d1ca4f5aeab3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf29e030892d44e1ad4d1ca4f5aeab3b already in progress: creating tablet
I20251029 08:52:11.891716 1176 tablet_replica.cc:333] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.891940 1176 raft_consensus.cc:2243] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.892182 1176 raft_consensus.cc:2272] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.892714 1183 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:11.897574 1183 tablet_bootstrap.cc:654] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:11.899556 1176 ts_tablet_manager.cc:1916] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.911345 1176 ts_tablet_manager.cc:1929] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:11.911540 1176 log.cc:1199] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aea79bd4e6844b7abac623824cabbeeb
I20251029 08:52:11.911952 1176 ts_tablet_manager.cc:1950] T aea79bd4e6844b7abac623824cabbeeb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:11.916556 1179 tablet_replica.cc:333] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:11.916582 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 425) took 1623 ms. Trace:
I20251029 08:52:11.916698 1176 rpcz_store.cc:276] 1029 08:52:10.292716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.303122 (+ 10406us) service_pool.cc:225] Handling call
1029 08:52:11.891650 (+1588528us) ts_tablet_manager.cc:1151] Deleting tablet aea79bd4e6844b7abac623824cabbeeb
1029 08:52:11.891660 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.906183 (+ 14523us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.911206 (+ 5023us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.916514 (+ 5308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576133}
I20251029 08:52:11.917737 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a20b64ff177241309e5d59e84f0bafa2 (table default.Table_9f8030a3d166447fbd35405c27dc35ec [id=478a20ec9fb6458d9ebeb17978e78703]) successfully deleted
W20251029 08:52:11.918083 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 19fdc381a0d949b0b93cef399daa3021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fdc381a0d949b0b93cef399daa3021 already in progress: creating tablet
I20251029 08:52:11.918570 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 424) took 1792 ms. Trace:
I20251029 08:52:11.918694 1176 rpcz_store.cc:276] 1029 08:52:10.126034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.302341 (+176307us) service_pool.cc:225] Handling call
1029 08:52:11.916992 (+1614651us) ts_tablet_manager.cc:1151] Deleting tablet 14b665b63a7341d699d499ffc8cfd9f7
1029 08:52:11.917004 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.918507 (+ 1503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595913}
I20251029 08:52:11.919545 1176 tablet_replica.cc:333] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:11.919690 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 14b665b63a7341d699d499ffc8cfd9f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b665b63a7341d699d499ffc8cfd9f7 already in progress: creating tablet
I20251029 08:52:11.919761 1176 raft_consensus.cc:2243] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.919962 1176 raft_consensus.cc:2272] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.921839 1176 ts_tablet_manager.cc:1916] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.936488 1176 ts_tablet_manager.cc:1929] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:11.936663 1176 log.cc:1199] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/fa8884bb8e64474d936c8825d1ef3545
W20251029 08:52:11.940469 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c68edbe3428b41579ae051abdb61b40d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:11.942963 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet a2947083e3e447d5addae5effd186eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.943527 1176 ts_tablet_manager.cc:1950] T fa8884bb8e64474d936c8825d1ef3545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:11.945664 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cdcc9a441c6a4d82a58262f4fdb3f00d consensus state has no leader
I20251029 08:52:11.950968 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 438) took 1650 ms. Trace:
I20251029 08:52:11.951848 1176 rpcz_store.cc:276] 1029 08:52:10.300950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.495615 (+194665us) service_pool.cc:225] Handling call
1029 08:52:11.919449 (+1423834us) ts_tablet_manager.cc:1151] Deleting tablet fa8884bb8e64474d936c8825d1ef3545
1029 08:52:11.919460 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.929487 (+ 10027us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.936318 (+ 6831us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:11.950912 (+ 14594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1423585}
I20251029 08:52:11.952834 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet fa8884bb8e64474d936c8825d1ef3545 (table default.Table_d3b4e5bde9434116b192d820a909c7b1 [id=064ab59757e94a9089fed2c7b3425476]) successfully deleted
I20251029 08:52:11.953436 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 439) took 1652 ms. Trace:
I20251029 08:52:11.954031 1176 rpcz_store.cc:276] 1029 08:52:10.301041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.495885 (+194844us) service_pool.cc:225] Handling call
1029 08:52:11.953217 (+1457332us) ts_tablet_manager.cc:1151] Deleting tablet ba1bcb3f6c134de2b322696501927a16
1029 08:52:11.953255 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.953396 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457192}
W20251029 08:52:11.954885 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ba1bcb3f6c134de2b322696501927a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba1bcb3f6c134de2b322696501927a16 already in progress: creating tablet
I20251029 08:52:11.955376 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 437) took 1654 ms. Trace:
I20251029 08:52:11.955830 1176 rpcz_store.cc:276] 1029 08:52:10.300829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.470253 (+169424us) service_pool.cc:225] Handling call
1029 08:52:11.955229 (+1484976us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:11.955240 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.955336 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439071}
W20251029 08:52:11.957679 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5ac1af8b78a4d94b0e3da14ebfffd92 already in progress: creating tablet
I20251029 08:52:11.958113 1176 tablet_replica.cc:333] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:11.959086 1176 raft_consensus.cc:2243] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:11.959913 1734 raft_consensus.cc:493] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:11.960119 1734 raft_consensus.cc:515] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:11.960853 1734 leader_election.cc:290] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:11.960907 1176 raft_consensus.cc:2272] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:11.961727 1176 ts_tablet_manager.cc:1916] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:11.962777 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet eeabf8f23751447aa9af02363c40361d
I20251029 08:52:11.964289 761 tablet_service.cc:1552] Processing DeleteTablet for tablet eeabf8f23751447aa9af02363c40361d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:11.964010 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet eeabf8f23751447aa9af02363c40361d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:11.966652 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 57c0ad83e3f54150850b6617a1781382 (DEFAULT_TABLE table=default.Table_7c15ab0db646407ab8460031c72ab852 [id=133929f2584345a98d53db9c2e42103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.967428 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c0ad83e3f54150850b6617a1781382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.980365 1178 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:11.980638 1178 ts_tablet_manager.cc:1403] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.187s user 0.009s sys 0.005s
I20251029 08:52:11.983976 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 (DEFAULT_TABLE table=default.Table_82984426f7eb4edf9c0cab2e8544d1dc [id=26985b122568423d81d46708fe76d64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.985033 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6691e1944ae947b49e4431f31ba9ae41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.985801 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec00cd64dc247da8442b6ba197f370a (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.986207 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec00cd64dc247da8442b6ba197f370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:11.986730 761 tablet_service.cc:1505] Processing CreateTablet for tablet 57c0ad83e3f54150850b6617a1781382 (DEFAULT_TABLE table=default.Table_7c15ab0db646407ab8460031c72ab852 [id=133929f2584345a98d53db9c2e42103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.989661 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cf906c73dc84403bcc3dece74a822bd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:11.991200 889 tablet_service.cc:1552] Processing DeleteTablet for tablet b1aaf233eba2461f9182e094a0115a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.991478 902 tablet_service.cc:1552] Processing DeleteTablet for tablet eeabf8f23751447aa9af02363c40361d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.991902 893 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec00cd64dc247da8442b6ba197f370a (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.994181 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.994925 890 tablet_service.cc:1505] Processing CreateTablet for tablet 57c0ad83e3f54150850b6617a1781382 (DEFAULT_TABLE table=default.Table_7c15ab0db646407ab8460031c72ab852 [id=133929f2584345a98d53db9c2e42103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.995421 897 tablet_service.cc:1505] Processing CreateTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 (DEFAULT_TABLE table=default.Table_82984426f7eb4edf9c0cab2e8544d1dc [id=26985b122568423d81d46708fe76d64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:11.996510 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:11.998157 892 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:11.999027 922 raft_consensus.cc:2468] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:11.999675 996 leader_election.cc:304] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:12.000365 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6017008571b64d0cb74edcd9f66e78a0" } modify_external_catalogs: true
I20251029 08:52:12.000659 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6017008571b64d0cb74edcd9f66e78a0" } modify_external_catalogs: true
I20251029 08:52:12.000824 1715 raft_consensus.cc:493] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.001019 1715 raft_consensus.cc:515] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.002295 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_69a998c83a8c499d86a9931423a1c53f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.003098 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a998c83a8c499d86a9931423a1c53f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.003693 1715 leader_election.cc:290] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:12.008700 1177 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:12.013996 1175 ts_tablet_manager.cc:1929] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:12.014617 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec00cd64dc247da8442b6ba197f370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.015175 1177 ts_tablet_manager.cc:1403] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.196s user 0.002s sys 0.002s
I20251029 08:52:12.016932 1177 raft_consensus.cc:359] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.018237 1177 raft_consensus.cc:385] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.018402 1177 raft_consensus.cc:740] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.018935 1715 raft_consensus.cc:493] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.019490 1177 consensus_queue.cc:260] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.020679 1177 ts_tablet_manager.cc:1434] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 08:52:12.020828 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6691e1944ae947b49e4431f31ba9ae41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.023133 588 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 3bad3607bb874c6b92f319082ac3bf13
I20251029 08:52:12.023294 1715 raft_consensus.cc:515] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.024009 1715 leader_election.cc:290] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:12.024394 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c0ad83e3f54150850b6617a1781382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.029058 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0a20ef9c732a4c6bb45224c8a4454cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.029886 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a20ef9c732a4c6bb45224c8a4454cf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.031642 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_23cbf2fd266c45efbdb3447e52d13f9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.032327 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23cbf2fd266c45efbdb3447e52d13f9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.033917 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3fb78e6a50342688581081eb505f8ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.034526 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3fb78e6a50342688581081eb505f8ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.037521 1177 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:12.041563 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3762da0232e4c168433f80930414895" } new_table_name: "default.Table_cb859e39a393451a8ec8ff23936fc7ee" modify_external_catalogs: true
I20251029 08:52:12.053346 1175 log.cc:1199] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4b55ae84c8d441459a5931abda471f6c
I20251029 08:52:12.058104 763 tablet_service.cc:1505] Processing CreateTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 (DEFAULT_TABLE table=default.Table_82984426f7eb4edf9c0cab2e8544d1dc [id=26985b122568423d81d46708fe76d64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.059155 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec00cd64dc247da8442b6ba197f370a (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.059522 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c0ad83e3f54150850b6617a1781382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.061057 1178 raft_consensus.cc:359] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.061269 1178 raft_consensus.cc:385] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:12.061759 861 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a57b1dbbdf854bc6a6e1b5dc88b63bff
W20251029 08:52:12.062028 861 leader_election.cc:343] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2e96cbdc4b7845b3823617a8e9a8c8b2
I20251029 08:52:12.062543 1178 raft_consensus.cc:740] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.062814 1178 consensus_queue.cc:260] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.063412 1178 ts_tablet_manager.cc:1434] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20251029 08:52:12.063782 1178 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:12.065603 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec00cd64dc247da8442b6ba197f370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.066645 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6691e1944ae947b49e4431f31ba9ae41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.071980 1176 ts_tablet_manager.cc:1929] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.072276 1176 log.cc:1199] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:12.072751 1176 ts_tablet_manager.cc:1950] T add0605edf5c4511ae4c8d9cb7f0babb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.074033 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 440) took 1759 ms. Trace:
I20251029 08:52:12.074414 1176 rpcz_store.cc:276] 1029 08:52:10.314240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.497428 (+183188us) service_pool.cc:225] Handling call
1029 08:52:11.958034 (+1460606us) ts_tablet_manager.cc:1151] Deleting tablet add0605edf5c4511ae4c8d9cb7f0babb
1029 08:52:11.958046 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.969946 (+ 11900us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.071831 (+101885us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.073972 (+ 2141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433311}
I20251029 08:52:12.074990 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 447) took 1697 ms. Trace:
I20251029 08:52:12.075148 1176 rpcz_store.cc:276] 1029 08:52:10.377023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.524746 (+147723us) service_pool.cc:225] Handling call
1029 08:52:12.074813 (+1550067us) ts_tablet_manager.cc:1151] Deleting tablet 7a355c68ecfb45ff8c396873d91268a8
1029 08:52:12.074825 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.074938 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549914}
I20251029 08:52:12.076725 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet add0605edf5c4511ae4c8d9cb7f0babb (table default.Table_3eda8a3eed924142880c9b465eb759b9 [id=edda8e15b59b472886bb1eba8fec8739]) successfully deleted
I20251029 08:52:12.076934 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 436) took 1776 ms. Trace:
W20251029 08:52:12.077100 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7a355c68ecfb45ff8c396873d91268a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a355c68ecfb45ff8c396873d91268a8 already in progress: creating tablet
I20251029 08:52:12.077031 1176 rpcz_store.cc:276] 1029 08:52:10.300558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.469799 (+169241us) service_pool.cc:225] Handling call
1029 08:52:12.076765 (+1606966us) ts_tablet_manager.cc:1151] Deleting tablet 2f40aafe7dc94851a67abd84d80d7d7b
1029 08:52:12.076776 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.076882 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548442}
W20251029 08:52:12.077466 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2f40aafe7dc94851a67abd84d80d7d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f40aafe7dc94851a67abd84d80d7d7b already in progress: creating tablet
I20251029 08:52:12.078176 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 448) took 1695 ms. Trace:
I20251029 08:52:12.083043 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3e9067e3526a433e8b4d64a6a5b739a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.083657 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e9067e3526a433e8b4d64a6a5b739a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.084028 1178 tablet_bootstrap.cc:654] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.085086 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_10573b6a1c074383b5808291ca1505ed" } modify_external_catalogs: true
I20251029 08:52:12.085255 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_10573b6a1c074383b5808291ca1505ed" } modify_external_catalogs: true
I20251029 08:52:12.085776 595 master_service.cc:954] ReplaceTablet: received request to replace tablet c7024e6eb4124109ad1575d216739fe2 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:12.086645 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_87c02133f89043fb86878e61cfe8d96d" } new_table_name: "default.Table_e9b105d36aed4a0da8586038f56dec12" modify_external_catalogs: true
I20251029 08:52:12.088270 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e65b3b1c8d14743aca1465d95ace95f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:12.088933 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e37db8e2538e4047b9bb40e72d99c616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.089308 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:12.089442 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e37db8e2538e4047b9bb40e72d99c616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.090950 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_54c3fb69aa194818946513e7b678f20a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.091089 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ea2ca0e66f9c40d1971ec171d61737fb" } modify_external_catalogs: true
I20251029 08:52:12.091326 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ea2ca0e66f9c40d1971ec171d61737fb" } modify_external_catalogs: true
W20251029 08:52:12.092706 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c3fb69aa194818946513e7b678f20a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.093855 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4deeec492a2545ca86583c6ea899a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.093998 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_317871afd272485a9a1789cf2f56a94f" } modify_external_catalogs: true
I20251029 08:52:12.094190 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_317871afd272485a9a1789cf2f56a94f" } modify_external_catalogs: true
I20251029 08:52:12.096076 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ddf59288eaf146a0a467f47f4911d920" } new_table_name: "default.Table_7e050a99735b4a27a96633bad5587e4b" modify_external_catalogs: true
I20251029 08:52:12.098819 1175 ts_tablet_manager.cc:1950] T 4b55ae84c8d441459a5931abda471f6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:12.078271 1176 rpcz_store.cc:276] 1029 08:52:10.383188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.524915 (+141727us) service_pool.cc:225] Handling call
1029 08:52:12.077995 (+1553080us) ts_tablet_manager.cc:1151] Deleting tablet 7b5051ff776e4975b917db0393a1f945
1029 08:52:12.078005 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.078110 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521362}
W20251029 08:52:12.105324 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 105698us
W20251029 08:52:12.105880 996 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:12.106535 1734 raft_consensus.cc:2804] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:12.106662 1734 raft_consensus.cc:493] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:12.106784 1734 raft_consensus.cc:3060] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.107025 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 463) took 1540 ms. Trace:
I20251029 08:52:12.107134 1176 rpcz_store.cc:276] 1029 08:52:10.567420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.850528 (+283108us) service_pool.cc:225] Handling call
1029 08:52:12.105519 (+1254991us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:12.105534 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.106959 (+ 1425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254676}
W20251029 08:52:12.108590 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7b5051ff776e4975b917db0393a1f945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b5051ff776e4975b917db0393a1f945 already in progress: creating tablet
W20251029 08:52:12.108911 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 657cdaad97e24611a997e26a6293ea5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657cdaad97e24611a997e26a6293ea5e already in progress: creating tablet
I20251029 08:52:12.108986 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 464) took 1536 ms. Trace:
I20251029 08:52:12.109078 1176 rpcz_store.cc:276] 1029 08:52:10.572848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.850877 (+278029us) service_pool.cc:225] Handling call
1029 08:52:12.108803 (+1257926us) ts_tablet_manager.cc:1151] Deleting tablet b7da7e08b74f4a9aacedada42e839a11
1029 08:52:12.108818 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.108929 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256741}
W20251029 08:52:12.109777 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b7da7e08b74f4a9aacedada42e839a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7da7e08b74f4a9aacedada42e839a11 already in progress: creating tablet
I20251029 08:52:12.110752 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 465) took 1530 ms. Trace:
I20251029 08:52:12.110854 1176 rpcz_store.cc:276] 1029 08:52:10.580442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.852110 (+271668us) service_pool.cc:225] Handling call
1029 08:52:12.110580 (+1258470us) ts_tablet_manager.cc:1151] Deleting tablet aa411391c63146e7acb8b4c6edbe12a0
1029 08:52:12.110591 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.110697 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258176}
W20251029 08:52:12.111304 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aa411391c63146e7acb8b4c6edbe12a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa411391c63146e7acb8b4c6edbe12a0 already in progress: creating tablet
I20251029 08:52:12.112277 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a79a91d787244124b45b7989b3c08c08" } new_table_name: "default.Table_5f0752745f5b45c7a3e2be18e3fac5a9" modify_external_catalogs: true
I20251029 08:52:12.112977 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 466) took 1524 ms. Trace:
I20251029 08:52:12.113070 1176 rpcz_store.cc:276] 1029 08:52:10.588134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.858714 (+270580us) service_pool.cc:225] Handling call
1029 08:52:12.112812 (+1254098us) ts_tablet_manager.cc:1151] Deleting tablet aa4f97075b3941aaa7cda3f9413e867f
1029 08:52:12.112823 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.112919 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252159}
W20251029 08:52:12.113638 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aa4f97075b3941aaa7cda3f9413e867f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa4f97075b3941aaa7cda3f9413e867f already in progress: creating tablet
W20251029 08:52:12.114827 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bfa85dfcdad46d3a7cbc7150c3b625f consensus state has no leader
I20251029 08:52:12.115235 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_13df179414624614b1663578c3d515be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.116390 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13df179414624614b1663578c3d515be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.116632 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 467) took 1506 ms. Trace:
I20251029 08:52:12.116731 1176 rpcz_store.cc:276] 1029 08:52:10.610533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.860698 (+250165us) service_pool.cc:225] Handling call
1029 08:52:12.116410 (+1255712us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:12.116423 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.116570 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255462}
I20251029 08:52:12.116732 1734 raft_consensus.cc:515] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:12.117187 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aafa846cb2144cec9ce4b28e61d91961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafa846cb2144cec9ce4b28e61d91961 already in progress: creating tablet
I20251029 08:52:12.117599 1741 raft_consensus.cc:493] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.118433 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cf906c73dc84403bcc3dece74a822bd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:12.118156 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 368) took 2870 ms. Trace:
I20251029 08:52:12.119270 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 474) took 1396 ms. Trace:
I20251029 08:52:12.119387 923 raft_consensus.cc:3060] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.119385 1176 rpcz_store.cc:276] 1029 08:52:10.723024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.946820 (+223796us) service_pool.cc:225] Handling call
1029 08:52:12.119094 (+1172274us) ts_tablet_manager.cc:1151] Deleting tablet 39fc9612689b46d498d6092bd552da4e
1029 08:52:12.119105 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.119210 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171920}
W20251029 08:52:12.119858 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 39fc9612689b46d498d6092bd552da4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39fc9612689b46d498d6092bd552da4e already in progress: creating tablet
W20251029 08:52:12.123216 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e8c4d793f10147fba5e4f44474575bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8c4d793f10147fba5e4f44474575bf0 already in progress: creating tablet
I20251029 08:52:12.123519 923 raft_consensus.cc:2468] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:12.124187 1715 raft_consensus.cc:493] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:12.124337 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 351047c725f8488abbd074bd40790b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351047c725f8488abbd074bd40790b48 already in progress: creating tablet
I20251029 08:52:12.124508 1715 raft_consensus.cc:515] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.125295 1715 leader_election.cc:290] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:12.126380 861 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 64f73597d1764540bcaf472d3e890ee8
W20251029 08:52:12.127076 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9907d044daa145168417371c2bd66544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9907d044daa145168417371c2bd66544 already in progress: creating tablet
W20251029 08:52:12.128708 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 43c3074caee74069a8a7f1b87721d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43c3074caee74069a8a7f1b87721d726 already in progress: creating tablet
I20251029 08:52:12.129140 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:12.130107 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ab04114f072046e2a09d1d5559571e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.130939 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab04114f072046e2a09d1d5559571e7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.128988 1175 rpcz_store.cc:276] 1029 08:52:09.233539 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.233959 (+ 420us) service_pool.cc:225] Handling call
1029 08:52:11.671360 (+2437401us) ts_tablet_manager.cc:1151] Deleting tablet 4b55ae84c8d441459a5931abda471f6c
1029 08:52:11.671372 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:11.825128 (+153756us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.013821 (+188693us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.104394 (+ 90573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434499}
I20251029 08:52:12.132256 1176 tablet_replica.cc:333] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:12.132472 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 19fdc381a0d949b0b93cef399daa3021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fdc381a0d949b0b93cef399daa3021 already in progress: creating tablet
I20251029 08:52:12.132485 1734 leader_election.cc:290] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:12.132807 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e3c24c0d32f74435bd9d28f2415e2c47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c24c0d32f74435bd9d28f2415e2c47 already in progress: creating tablet
W20251029 08:52:12.133039 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet af38662420d14ac1a7cb5fd03fc1b9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af38662420d14ac1a7cb5fd03fc1b9d8 already in progress: creating tablet
I20251029 08:52:12.132664 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:12.133301 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d50e2fdfc0e64516a41c90f5a56f4c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d50e2fdfc0e64516a41c90f5a56f4c89 already in progress: creating tablet
I20251029 08:52:12.133656 1176 raft_consensus.cc:2243] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.134234 996 leader_election.cc:304] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:12.134624 1176 raft_consensus.cc:2272] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.135188 1176 ts_tablet_manager.cc:1916] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.136500 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.139386 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4b55ae84c8d441459a5931abda471f6c (table default.Table_c630c3859f114c0d871851d8f87a6544 [id=fa54a881d58045fdb998132a18e1c890]) successfully deleted
I20251029 08:52:12.140321 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c7024e6eb4124109ad1575d216739fe2
I20251029 08:52:12.140980 595 catalog_manager.cc:6719] ReplaceTablet: tablet c7024e6eb4124109ad1575d216739fe2 deleted and replaced by tablet 90360621ce124475af2f12060471f63c
I20251029 08:52:12.141388 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_175a44baba964b30b0fd67d4cd921479" } new_table_name: "default.Table_f1d87583fc3b4975b8dc200489ff22ce" modify_external_catalogs: true
I20251029 08:52:12.142921 751 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.148931 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.149183 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:12.153688 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7046bb1fcdd34d308aecddfdf0752911 consensus state has no leader
I20251029 08:52:12.154255 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b37a9d683d684befa22f0c33ef953739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.155055 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b37a9d683d684befa22f0c33ef953739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.155066 1176 ts_tablet_manager.cc:1929] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.155196 1176 log.cc:1199] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/be706771a95e45e5a20a15614eadfe1a
I20251029 08:52:12.155550 1176 ts_tablet_manager.cc:1950] T be706771a95e45e5a20a15614eadfe1a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.151736 1741 raft_consensus.cc:515] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.157359 892 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.158866 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 366) took 2910 ms. Trace:
I20251029 08:52:12.159632 892 tablet_service.cc:1505] Processing CreateTablet for tablet e322478c9ba64c2a95a2c299123926dd (DEFAULT_TABLE table=default.Table_620a010c6d044d05af5e6a82066c21c3 [id=940d68f536cb449997a16cd3a2e6a0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.160132 1177 tablet_bootstrap.cc:654] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.162981 1175 rpcz_store.cc:276] 1029 08:52:09.233329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.234002 (+ 673us) service_pool.cc:225] Handling call
1029 08:52:12.138200 (+2904198us) ts_tablet_manager.cc:1151] Deleting tablet ea13e9ea2c11479ea86e3fd7f2a9e59c
1029 08:52:12.138212 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.144162 (+ 5950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899199}
W20251029 08:52:12.164646 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea13e9ea2c11479ea86e3fd7f2a9e59c already in progress: creating tablet
I20251029 08:52:12.167212 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet be706771a95e45e5a20a15614eadfe1a (table default.Table_a69aba63f41c4596a1017f38f6ed6532 [id=e20293dc36d54acf9ee6f9bf7ac55834]) successfully deleted
W20251029 08:52:12.168946 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 840ff254244f43a18679232279e02d1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840ff254244f43a18679232279e02d1f already in progress: creating tablet
I20251029 08:52:12.169021 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf" } modify_external_catalogs: true
I20251029 08:52:12.169296 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf" } modify_external_catalogs: true
W20251029 08:52:12.170219 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 608b7095d42941ccb0c4eb57f6754e74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608b7095d42941ccb0c4eb57f6754e74 already in progress: creating tablet
I20251029 08:52:12.170467 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e322478c9ba64c2a95a2c299123926dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.171319 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a31847928db44bd4b1ebd50c5130065f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31847928db44bd4b1ebd50c5130065f already in progress: creating tablet
I20251029 08:52:12.171942 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2ca466714c774fc9982ef145c6dd1571
W20251029 08:52:12.172340 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 627bc8ff99394fa2abcca15d62e5ea10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 627bc8ff99394fa2abcca15d62e5ea10 already in progress: creating tablet
I20251029 08:52:12.173331 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ca466714c774fc9982ef145c6dd1571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:12.173638 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e0fa089fce564af4b03aa323cec6cbf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fa089fce564af4b03aa323cec6cbf6 already in progress: creating tablet
W20251029 08:52:12.174930 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d5bb0fd4c62d47a2b10719ebd55851c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bb0fd4c62d47a2b10719ebd55851c0 already in progress: creating tablet
W20251029 08:52:12.177266 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 757e34137e884d5ca5cd88fdd03244d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757e34137e884d5ca5cd88fdd03244d8 already in progress: creating tablet
W20251029 08:52:12.179174 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 49d9c29b93a643ec837169a20c59de33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9c29b93a643ec837169a20c59de33 already in progress: creating tablet
I20251029 08:52:12.179848 1687 raft_consensus.cc:493] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.180068 1687 raft_consensus.cc:515] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:12.180510 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 652903cbbdb042d780e7d71fc2c0ab4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652903cbbdb042d780e7d71fc2c0ab4d already in progress: creating tablet
I20251029 08:52:12.180784 1687 leader_election.cc:290] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.181440 1176 tablet_replica.cc:333] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.181744 1176 raft_consensus.cc:2243] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.181918 1176 raft_consensus.cc:2272] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.182528 1176 ts_tablet_manager.cc:1916] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.185977 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abc307b14aff4c5e937a5151aab1dfb9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:12.186766 718 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.188055 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 365) took 2947 ms. Trace:
I20251029 08:52:12.188244 1175 rpcz_store.cc:276] 1029 08:52:09.233178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.233978 (+ 800us) service_pool.cc:225] Handling call
1029 08:52:12.167520 (+2933542us) ts_tablet_manager.cc:1151] Deleting tablet 74fe974a368a4afe8a5da6dfd1036400
1029 08:52:12.179186 (+ 11666us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.181041 (+ 1855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2928417}
W20251029 08:52:12.188949 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 74fe974a368a4afe8a5da6dfd1036400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fe974a368a4afe8a5da6dfd1036400 already in progress: creating tablet
I20251029 08:52:12.190889 1175 tablet_replica.cc:333] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:12.191145 1175 raft_consensus.cc:2243] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:12.192925 1175 raft_consensus.cc:2272] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.194741 1175 ts_tablet_manager.cc:1916] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.199266 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ca466714c774fc9982ef145c6dd1571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.201457 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a7a673ba4fd4abcbfc3493c38dcc257 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:12.207340 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ca466714c774fc9982ef145c6dd1571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.207747 751 tablet_service.cc:1505] Processing CreateTablet for tablet e322478c9ba64c2a95a2c299123926dd (DEFAULT_TABLE table=default.Table_620a010c6d044d05af5e6a82066c21c3 [id=940d68f536cb449997a16cd3a2e6a0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.208223 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e322478c9ba64c2a95a2c299123926dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.209203 763 tablet_service.cc:1505] Processing CreateTablet for tablet 57804eef0fee4b2db02b1acb75fc813d (DEFAULT_TABLE table=default.Table_752b2204770149368a72732e2da51c1a [id=12e8bf15ceff40fb923a43e34ddc8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.209440 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:12.209597 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57804eef0fee4b2db02b1acb75fc813d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.210386 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.211345 761 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f (DEFAULT_TABLE table=default.Table_cb859e39a393451a8ec8ff23936fc7ee [id=cfb11fa969b84f75891152a2028c7e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.211756 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.219969 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abc307b14aff4c5e937a5151aab1dfb9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:12.220613 720 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:12.220733 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 consensus state has no leader
I20251029 08:52:12.220763 720 leader_election.cc:304] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:12.221136 1178 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
W20251029 08:52:12.221181 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fafe9a0731b48b3a60032d94623681d consensus state has no leader
I20251029 08:52:12.221267 1687 raft_consensus.cc:2749] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.221364 1178 ts_tablet_manager.cc:1403] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.158s user 0.010s sys 0.018s
I20251029 08:52:12.222242 1178 raft_consensus.cc:359] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.223297 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71ba5e1248dc4d4a8a4d7bb05b661a3f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:12.223678 1178 raft_consensus.cc:385] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.223814 1178 raft_consensus.cc:740] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.224177 1178 consensus_queue.cc:260] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.224478 923 raft_consensus.cc:2468] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:12.224913 1178 ts_tablet_manager.cc:1434] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251029 08:52:12.225534 890 tablet_service.cc:1505] Processing CreateTablet for tablet 57804eef0fee4b2db02b1acb75fc813d (DEFAULT_TABLE table=default.Table_752b2204770149368a72732e2da51c1a [id=12e8bf15ceff40fb923a43e34ddc8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.226591 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e6415f97a9d64fc996f4e571e61aba62" } modify_external_catalogs: true
I20251029 08:52:12.226701 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57804eef0fee4b2db02b1acb75fc813d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.226796 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e6415f97a9d64fc996f4e571e61aba62" } modify_external_catalogs: true
I20251029 08:52:12.227797 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:12.231557 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5ae1f06e4289499eb646135fd2430a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.231573 761 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:12.232179 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ae1f06e4289499eb646135fd2430a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.233852 1178 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:12.235340 893 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f (DEFAULT_TABLE table=default.Table_cb859e39a393451a8ec8ff23936fc7ee [id=cfb11fa969b84f75891152a2028c7e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.235721 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.239387 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.239887 1047 tablet_service.cc:1505] Processing CreateTablet for tablet e322478c9ba64c2a95a2c299123926dd (DEFAULT_TABLE table=default.Table_620a010c6d044d05af5e6a82066c21c3 [id=940d68f536cb449997a16cd3a2e6a0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.240420 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e322478c9ba64c2a95a2c299123926dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.242093 1178 tablet_bootstrap.cc:654] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.252368 1178 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:12.252952 1178 ts_tablet_manager.cc:1403] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.005s
W20251029 08:52:12.253376 861 consensus_peers.cc:597] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aea79bd4e6844b7abac623824cabbeeb. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:12.253918 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f (DEFAULT_TABLE table=default.Table_cb859e39a393451a8ec8ff23936fc7ee [id=cfb11fa969b84f75891152a2028c7e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.254014 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a57b1dbbdf854bc6a6e1b5dc88b63bff" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.254402 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfa85dfcdad46d3a7cbc7150c3b625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.255065 1178 raft_consensus.cc:359] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.255249 1178 raft_consensus.cc:385] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.248944 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e96cbdc4b7845b3823617a8e9a8c8b2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.256187 1178 raft_consensus.cc:740] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.256593 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 57804eef0fee4b2db02b1acb75fc813d (DEFAULT_TABLE table=default.Table_752b2204770149368a72732e2da51c1a [id=12e8bf15ceff40fb923a43e34ddc8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.256629 1178 consensus_queue.cc:260] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.253649 1741 leader_election.cc:290] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:12.256186 998 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4cf906c73dc84403bcc3dece74a822bd
I20251029 08:52:12.257596 996 leader_election.cc:304] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:12.258050 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.258391 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64f73597d1764540bcaf472d3e890ee8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.258882 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57804eef0fee4b2db02b1acb75fc813d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.261615 998 leader_election.cc:343] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4cf906c73dc84403bcc3dece74a822bd
W20251029 08:52:12.262004 998 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:12.264626 1178 ts_tablet_manager.cc:1434] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251029 08:52:12.266688 761 tablet_service.cc:1505] Processing CreateTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.267237 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bf5d693be240f18bacccc81a6ea5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.267536 1741 raft_consensus.cc:2804] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:12.273176 1178 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:12.278017 1178 tablet_bootstrap.cc:654] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.292902 1176 ts_tablet_manager.cc:1929] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.294090 892 tablet_service.cc:1505] Processing CreateTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.294684 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bf5d693be240f18bacccc81a6ea5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.294996 1176 log.cc:1199] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/94738d48801a4fb4a7ad1f5c4022afe1
I20251029 08:52:12.295492 1176 ts_tablet_manager.cc:1950] T 94738d48801a4fb4a7ad1f5c4022afe1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.301707 1708 raft_consensus.cc:493] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:12.302011 1708 raft_consensus.cc:515] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:12.303375 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 94738d48801a4fb4a7ad1f5c4022afe1 (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:12.304607 1675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:12.304873 1675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
W20251029 08:52:12.307673 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fc04616301354cfe873c825a9f5d7ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc04616301354cfe873c825a9f5d7ab6 already in progress: creating tablet
I20251029 08:52:12.307852 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ffc61d294dca458aba887c7e9614a720"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.308178 1176 tablet_replica.cc:333] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.308409 1176 raft_consensus.cc:2243] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.308642 1176 raft_consensus.cc:2272] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:12.308949 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffc61d294dca458aba887c7e9614a720 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.309283 1176 ts_tablet_manager.cc:1916] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.310389 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_44386c352f5e4b9795c8df047079ba09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.311085 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44386c352f5e4b9795c8df047079ba09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.314412 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8cc80787c54a42c4af323289ffd882e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.314772 1708 leader_election.cc:290] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
W20251029 08:52:12.315232 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cc80787c54a42c4af323289ffd882e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.320986 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9357468257554705827bd3bc1eee7bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.321743 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9357468257554705827bd3bc1eee7bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.323473 1687 raft_consensus.cc:493] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.323683 1687 raft_consensus.cc:515] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.324179 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c9cc46a4b97451594bc17d4eaf11972" } new_table_name: "default.Table_8d2ea6ac0509441282fd5b4a11217079" modify_external_catalogs: true
I20251029 08:52:12.324483 1687 leader_election.cc:290] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.325991 1178 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:12.327564 1178 ts_tablet_manager.cc:1403] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.014s
I20251029 08:52:12.302012 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37076 (request call id 0) took 3204 ms. Trace:
I20251029 08:52:12.328954 1178 raft_consensus.cc:359] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.332082 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8626877e3cfc4cb48f6caa92638b4917" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:12.332696 720 leader_election.cc:343] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.333349 1178 raft_consensus.cc:385] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.333505 1178 raft_consensus.cc:740] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.333823 1178 consensus_queue.cc:260] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.338145 1178 ts_tablet_manager.cc:1434] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:12.338528 1178 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:12.339553 1751 raft_consensus.cc:2804] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:12.339711 1751 raft_consensus.cc:493] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:12.340071 1751 raft_consensus.cc:3060] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.343504 1751 raft_consensus.cc:515] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.344112 1178 tablet_bootstrap.cc:654] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.357679 1176 ts_tablet_manager.cc:1929] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.359509 1179 raft_consensus.cc:2243] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:12.369611 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71ba5e1248dc4d4a8a4d7bb05b661a3f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:12.379550 923 raft_consensus.cc:3060] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.380199 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8626877e3cfc4cb48f6caa92638b4917" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:12.383601 1176 log.cc:1199] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/abc307b14aff4c5e937a5151aab1dfb9
W20251029 08:52:12.383638 718 leader_election.cc:343] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.383919 718 leader_election.cc:304] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:12.384608 1687 raft_consensus.cc:2749] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.384763 1176 ts_tablet_manager.cc:1950] T abc307b14aff4c5e937a5151aab1dfb9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.386896 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet abc307b14aff4c5e937a5151aab1dfb9 (table default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]) successfully deleted
W20251029 08:52:12.402733 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10baa0547180493f872a23614cf4c608 consensus state has no leader
I20251029 08:52:12.403105 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5104aa104b704bfea93e338b7efc8906" } modify_external_catalogs: true
I20251029 08:52:12.403321 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5104aa104b704bfea93e338b7efc8906" } modify_external_catalogs: true
I20251029 08:52:12.404120 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5e62c5fdfba04b7287b6ec49e5af68d1" } new_table_name: "default.Table_e7328a4d3185459186db060425ff7e13" modify_external_catalogs: true
I20251029 08:52:12.405565 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e5fa181849474a559701fdb8c18d398b
I20251029 08:52:12.406420 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fc047f205272431fa1ba26215c0013d1" } modify_external_catalogs: true
I20251029 08:52:12.406632 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fc047f205272431fa1ba26215c0013d1" } modify_external_catalogs: true
I20251029 08:52:12.407171 1177 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:12.407354 1177 ts_tablet_manager.cc:1403] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.371s user 0.005s sys 0.006s
I20251029 08:52:12.412672 751 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.414456 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.420521 761 tablet_service.cc:1505] Processing CreateTablet for tablet b5e1e3ce83b14817b2440370cb2fcabd (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.421192 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e1e3ce83b14817b2440370cb2fcabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.423751 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 072cf27b0743488c8166e03b5e61c545
I20251029 08:52:12.426239 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.427150 1759 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 08:52:12.429396 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0c14c63f74f74a18902a17a4d210a90d (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.429905 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14c63f74f74a18902a17a4d210a90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.432649 1176 tablet_replica.cc:333] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:12.432685 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3e65b3b1c8d14743aca1465d95ace95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e65b3b1c8d14743aca1465d95ace95f already in progress: creating tablet
I20251029 08:52:12.432880 1176 raft_consensus.cc:2243] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:12.432991 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c9bbf6401423411089c9070f3b03f1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bbf6401423411089c9070f3b03f1a7 already in progress: creating tablet
I20251029 08:52:12.433061 1176 raft_consensus.cc:2272] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:12.433269 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 633ace1413464e939e4e47782370e281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 633ace1413464e939e4e47782370e281 already in progress: creating tablet
I20251029 08:52:12.433284 763 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca466714c774fc9982ef145c6dd1571 (DEFAULT_TABLE table=default.Table_ea2ca0e66f9c40d1971ec171d61737fb [id=0a9b61d3c85941df84deaf0250a2a557]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:12.433521 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b1aaf233eba2461f9182e094a0115a4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1aaf233eba2461f9182e094a0115a4e already in progress: creating tablet
I20251029 08:52:12.433660 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca466714c774fc9982ef145c6dd1571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.433748 1176 ts_tablet_manager.cc:1916] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.436390 764 tablet_service.cc:1505] Processing CreateTablet for tablet fc78a202be244c3e9ad354bead9bfe52 (DEFAULT_TABLE table=default.Table_5038a1ad6f534f59a938f5b34e5e27db [id=6f7753d689454517b2ba34f007794f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.436811 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc78a202be244c3e9ad354bead9bfe52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.453639 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ff59f466b5804429bc09b24c36220b1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.455446 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff59f466b5804429bc09b24c36220b1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.458863 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.462287 923 raft_consensus.cc:2468] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:12.463284 888 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca466714c774fc9982ef145c6dd1571 (DEFAULT_TABLE table=default.Table_ea2ca0e66f9c40d1971ec171d61737fb [id=0a9b61d3c85941df84deaf0250a2a557]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.463944 1715 raft_consensus.cc:493] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.465866 896 tablet_service.cc:1505] Processing CreateTablet for tablet 0c14c63f74f74a18902a17a4d210a90d (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.466480 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca466714c774fc9982ef145c6dd1571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.467437 1177 raft_consensus.cc:359] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.468339 902 tablet_service.cc:1505] Processing CreateTablet for tablet fc78a202be244c3e9ad354bead9bfe52 (DEFAULT_TABLE table=default.Table_5038a1ad6f534f59a938f5b34e5e27db [id=6f7753d689454517b2ba34f007794f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.469357 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.472772 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bee86c1895544f8a9f972f5cc3662a3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.473484 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bee86c1895544f8a9f972f5cc3662a3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.478773 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 467 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:52:12.484336 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_61c7054042014cdb9fde5c728083e742" } modify_external_catalogs: true
I20251029 08:52:12.484599 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_61c7054042014cdb9fde5c728083e742" } modify_external_catalogs: true
I20251029 08:52:12.486085 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 467 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:12.488648 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6b5e334f84e049eea474e2587d954f23" } new_table_name: "default.Table_ba1277f42faa48bf9e570e5ad794055b" modify_external_catalogs: true
I20251029 08:52:12.495482 1451 leader_election.cc:304] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, 85c8cdf28bda444d851fb50588400173
I20251029 08:52:12.496896 1708 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251029 08:52:12.498282 998 consensus_peers.cc:597] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e2b19a227740a28ae601d77e62597b. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:12.498493 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 236202us
W20251029 08:52:12.498759 998 leader_election.cc:343] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:12.498924 998 leader_election.cc:304] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:12.500541 998 leader_election.cc:343] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:12.501313 1737 raft_consensus.cc:2749] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.501535 1737 raft_consensus.cc:493] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.501744 1737 raft_consensus.cc:515] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.502584 1737 leader_election.cc:290] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.507149 1715 raft_consensus.cc:515] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.507781 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14c63f74f74a18902a17a4d210a90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.508674 1177 raft_consensus.cc:385] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.510129 1176 ts_tablet_manager.cc:1929] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.510219 1183 tablet_bootstrap.cc:492] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:12.510340 1176 log.cc:1199] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/79a3ef8389944c32bba7b3e598824118
I20251029 08:52:12.510403 1183 ts_tablet_manager.cc:1403] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.618s user 0.113s sys 0.163s
I20251029 08:52:12.511514 1176 ts_tablet_manager.cc:1950] T 79a3ef8389944c32bba7b3e598824118 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.511200 1751 leader_election.cc:290] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.511293 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.512789 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 79a3ef8389944c32bba7b3e598824118 (table default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]) successfully deleted
I20251029 08:52:12.512882 1183 raft_consensus.cc:359] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.513067 1183 raft_consensus.cc:385] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.512854 996 leader_election.cc:304] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:12.513685 998 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a2947083e3e447d5addae5effd186eb8
I20251029 08:52:12.514042 893 tablet_service.cc:1505] Processing CreateTablet for tablet b5e1e3ce83b14817b2440370cb2fcabd (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.515357 1183 raft_consensus.cc:740] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.516155 1183 consensus_queue.cc:260] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.516942 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.517158 1751 raft_consensus.cc:2804] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:12.517408 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 (DEFAULT_TABLE table=default.Table_6ba990fb812943a5bd7c7588dbb684dd [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.519860 1044 tablet_service.cc:1505] Processing CreateTablet for tablet fc78a202be244c3e9ad354bead9bfe52 (DEFAULT_TABLE table=default.Table_5038a1ad6f534f59a938f5b34e5e27db [id=6f7753d689454517b2ba34f007794f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.520413 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc78a202be244c3e9ad354bead9bfe52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.522765 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc78a202be244c3e9ad354bead9bfe52. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.523293 861 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.523447 861 leader_election.cc:304] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:12.528963 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f5b52314139945dd9c2e6ec899bf06dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b52314139945dd9c2e6ec899bf06dc already in progress: creating tablet
W20251029 08:52:12.529760 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet eeabf8f23751447aa9af02363c40361d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeabf8f23751447aa9af02363c40361d already in progress: creating tablet
I20251029 08:52:12.329236 1525 rpcz_store.cc:276] 1029 08:52:09.097648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.098006 (+ 358us) service_pool.cc:225] Handling call
1029 08:52:09.635253 (+537247us) raft_consensus.cc:1471] Updating replica for 422 ops
1029 08:52:10.079476 (+444223us) raft_consensus.cc:1534] Early marking committed up to index 0
1029 08:52:10.079486 (+ 10us) raft_consensus.cc:1539] Triggering prepare for 422 ops
1029 08:52:10.080332 (+ 846us) write_op.cc:270] Start()
1029 08:52:10.080391 (+ 59us) write_op.cc:276] Timestamp: P: 1761727922043705 usec, L: 0
1029 08:52:10.363515 (+283124us) write_op.cc:270] Start()
1029 08:52:10.363554 (+ 39us) write_op.cc:276] Timestamp: P: 1761727922069029 usec, L: 0
1029 08:52:10.363781 (+ 227us) write_op.cc:270] Start()
1029 08:52:10.363821 (+ 40us) write_op.cc:276] Timestamp: P: 1761727922070073 usec, L: 0
1029 08:52:10.363966 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.363986 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922081392 usec, L: 0
1029 08:52:10.364153 (+ 167us) write_op.cc:270] Start()
1029 08:52:10.364174 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922089847 usec, L: 0
1029 08:52:10.364344 (+ 170us) write_op.cc:270] Start()
1029 08:52:10.364368 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922100783 usec, L: 0
1029 08:52:10.364531 (+ 163us) write_op.cc:270] Start()
1029 08:52:10.364576 (+ 45us) write_op.cc:276] Timestamp: P: 1761727922102650 usec, L: 0
1029 08:52:10.364713 (+ 137us) write_op.cc:270] Start()
1029 08:52:10.364730 (+ 17us) write_op.cc:276] Timestamp: P: 1761727922103790 usec, L: 0
1029 08:52:10.364885 (+ 155us) write_op.cc:270] Start()
1029 08:52:10.364904 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922106356 usec, L: 0
1029 08:52:10.365022 (+ 118us) write_op.cc:270] Start()
1029 08:52:10.365039 (+ 17us) write_op.cc:276] Timestamp: P: 1761727922151949 usec, L: 0
1029 08:52:10.365173 (+ 134us) write_op.cc:270] Start()
1029 08:52:10.365190 (+ 17us) write_op.cc:276] Timestamp: P: 1761727922258002 usec, L: 0
1029 08:52:10.365308 (+ 118us) write_op.cc:270] Start()
1029 08:52:10.365339 (+ 31us) write_op.cc:276] Timestamp: P: 1761727922271048 usec, L: 0
1029 08:52:10.365498 (+ 159us) write_op.cc:270] Start()
1029 08:52:10.365520 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922291350 usec, L: 0
1029 08:52:10.365657 (+ 137us) write_op.cc:270] Start()
1029 08:52:10.365677 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922366300 usec, L: 0
1029 08:52:10.365803 (+ 126us) write_op.cc:270] Start()
1029 08:52:10.365824 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922383162 usec, L: 0
1029 08:52:10.365965 (+ 141us) write_op.cc:270] Start()
1029 08:52:10.365985 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922394416 usec, L: 0
1029 08:52:10.366279 (+ 294us) write_op.cc:270] Start()
1029 08:52:10.366307 (+ 28us) write_op.cc:276] Timestamp: P: 1761727922396325 usec, L: 0
1029 08:52:10.366464 (+ 157us) write_op.cc:270] Start()
1029 08:52:10.366484 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922397492 usec, L: 0
1029 08:52:10.366631 (+ 147us) write_op.cc:270] Start()
1029 08:52:10.366653 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922427511 usec, L: 0
1029 08:52:10.366800 (+ 147us) write_op.cc:270] Start()
1029 08:52:10.369800 (+ 3000us) write_op.cc:276] Timestamp: P: 1761727922454468 usec, L: 0
1029 08:52:10.369976 (+ 176us) write_op.cc:270] Start()
1029 08:52:10.370000 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922465084 usec, L: 0
1029 08:52:10.370158 (+ 158us) write_op.cc:270] Start()
1029 08:52:10.370183 (+ 25us) write_op.cc:276] Timestamp: P: 1761727922475536 usec, L: 0
1029 08:52:10.370329 (+ 146us) write_op.cc:270] Start()
1029 08:52:10.370351 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922493299 usec, L: 0
1029 08:52:10.370483 (+ 132us) write_op.cc:270] Start()
1029 08:52:10.370502 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922536425 usec, L: 0
1029 08:52:10.370673 (+ 171us) write_op.cc:270] Start()
1029 08:52:10.370698 (+ 25us) write_op.cc:276] Timestamp: P: 1761727922542384 usec, L: 0
1029 08:52:10.370875 (+ 177us) write_op.cc:270] Start()
1029 08:52:10.370902 (+ 27us) write_op.cc:276] Timestamp: P: 1761727922543229 usec, L: 0
1029 08:52:10.371045 (+ 143us) write_op.cc:270] Start()
1029 08:52:10.371062 (+ 17us) write_op.cc:276] Timestamp: P: 1761727922544773 usec, L: 0
1029 08:52:10.371174 (+ 112us) write_op.cc:270] Start()
1029 08:52:10.371192 (+ 18us) write_op.cc:276] Timestamp: P: 1761727922545495 usec, L: 0
1029 08:52:10.371359 (+ 167us) write_op.cc:270] Start()
1029 08:52:10.371385 (+ 26us) write_op.cc:276] Timestamp: P: 1761727922546182 usec, L: 0
1029 08:52:10.371530 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.371551 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922546868 usec, L: 0
1029 08:52:10.371696 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.371717 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922547584 usec, L: 0
1029 08:52:10.371858 (+ 141us) write_op.cc:270] Start()
1029 08:52:10.371876 (+ 18us) write_op.cc:276] Timestamp: P: 1761727922560332 usec, L: 0
1029 08:52:10.372113 (+ 237us) write_op.cc:270] Start()
1029 08:52:10.372135 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922561002 usec, L: 0
1029 08:52:10.372284 (+ 149us) write_op.cc:270] Start()
1029 08:52:10.372306 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922575014 usec, L: 0
1029 08:52:10.372466 (+ 160us) write_op.cc:270] Start()
1029 08:52:10.372484 (+ 18us) write_op.cc:276] Timestamp: P: 1761727922583147 usec, L: 0
1029 08:52:10.372598 (+ 114us) write_op.cc:270] Start()
1029 08:52:10.372614 (+ 16us) write_op.cc:276] Timestamp: P: 1761727922587383 usec, L: 0
1029 08:52:10.372717 (+ 103us) write_op.cc:270] Start()
1029 08:52:10.372741 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922616774 usec, L: 0
1029 08:52:10.372843 (+ 102us) write_op.cc:270] Start()
1029 08:52:10.372859 (+ 16us) write_op.cc:276] Timestamp: P: 1761727922633347 usec, L: 0
1029 08:52:10.372977 (+ 118us) write_op.cc:270] Start()
1029 08:52:10.372995 (+ 18us) write_op.cc:276] Timestamp: P: 1761727922642337 usec, L: 0
1029 08:52:10.373134 (+ 139us) write_op.cc:270] Start()
1029 08:52:10.373153 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922653708 usec, L: 0
1029 08:52:10.373322 (+ 169us) write_op.cc:270] Start()
1029 08:52:10.373342 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922659277 usec, L: 0
1029 08:52:10.373493 (+ 151us) write_op.cc:270] Start()
1029 08:52:10.373513 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922660141 usec, L: 0
1029 08:52:10.373655 (+ 142us) write_op.cc:270] Start()
1029 08:52:10.373674 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922661965 usec, L: 0
1029 08:52:10.373807 (+ 133us) write_op.cc:270] Start()
1029 08:52:10.373830 (+ 23us) write_op.cc:276] Timestamp: P: 1761727922681648 usec, L: 0
1029 08:52:10.373979 (+ 149us) write_op.cc:270] Start()
1029 08:52:10.380209 (+ 6230us) write_op.cc:276] Timestamp: P: 1761727922683676 usec, L: 0
1029 08:52:10.380409 (+ 200us) write_op.cc:270] Start()
1029 08:52:10.380607 (+ 198us) write_op.cc:276] Timestamp: P: 1761727922685358 usec, L: 0
1029 08:52:10.380790 (+ 183us) write_op.cc:270] Start()
1029 08:52:10.380818 (+ 28us) write_op.cc:276] Timestamp: P: 1761727922690648 usec, L: 0
1029 08:52:10.380969 (+ 151us) write_op.cc:270] Start()
1029 08:52:10.380990 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922735372 usec, L: 0
1029 08:52:10.381132 (+ 142us) write_op.cc:270] Start()
1029 08:52:10.381154 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922773371 usec, L: 0
1029 08:52:10.381269 (+ 115us) write_op.cc:270] Start()
1029 08:52:10.381288 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922791712 usec, L: 0
1029 08:52:10.381697 (+ 409us) write_op.cc:270] Start()
1029 08:52:10.381721 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922794956 usec, L: 0
1029 08:52:10.381862 (+ 141us) write_op.cc:270] Start()
1029 08:52:10.381887 (+ 25us) write_op.cc:276] Timestamp: P: 1761727922811386 usec, L: 0
1029 08:52:10.382032 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.382050 (+ 18us) write_op.cc:276] Timestamp: P: 1761727922812164 usec, L: 0
1029 08:52:10.382331 (+ 281us) write_op.cc:270] Start()
1029 08:52:10.382358 (+ 27us) write_op.cc:276] Timestamp: P: 1761727922812800 usec, L: 0
1029 08:52:10.383111 (+ 753us) write_op.cc:270] Start()
1029 08:52:10.383140 (+ 29us) write_op.cc:276] Timestamp: P: 1761727922839793 usec, L: 0
1029 08:52:10.383301 (+ 161us) write_op.cc:270] Start()
1029 08:52:10.383343 (+ 42us) write_op.cc:276] Timestamp: P: 1761727922860432 usec, L: 0
1029 08:52:10.383493 (+ 150us) write_op.cc:270] Start()
1029 08:52:10.383516 (+ 23us) write_op.cc:276] Timestamp: P: 1761727922862032 usec, L: 0
1029 08:52:10.383650 (+ 134us) write_op.cc:270] Start()
1029 08:52:10.383674 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922879843 usec, L: 0
1029 08:52:10.383813 (+ 139us) write_op.cc:270] Start()
1029 08:52:10.383832 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922888951 usec, L: 0
1029 08:52:10.383948 (+ 116us) write_op.cc:270] Start()
1029 08:52:10.383987 (+ 39us) write_op.cc:276] Timestamp: P: 1761727922896084 usec, L: 0
1029 08:52:10.386043 (+ 2056us) write_op.cc:270] Start()
1029 08:52:10.386075 (+ 32us) write_op.cc:276] Timestamp: P: 1761727922904210 usec, L: 0
1029 08:52:10.386251 (+ 176us) write_op.cc:270] Start()
1029 08:52:10.386273 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922904912 usec, L: 0
1029 08:52:10.386437 (+ 164us) write_op.cc:270] Start()
1029 08:52:10.386459 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922905523 usec, L: 0
1029 08:52:10.386596 (+ 137us) write_op.cc:270] Start()
1029 08:52:10.386617 (+ 21us) write_op.cc:276] Timestamp: P: 1761727922906525 usec, L: 0
1029 08:52:10.386798 (+ 181us) write_op.cc:270] Start()
1029 08:52:10.386818 (+ 20us) write_op.cc:276] Timestamp: P: 1761727922968279 usec, L: 0
1029 08:52:10.386959 (+ 141us) write_op.cc:270] Start()
1029 08:52:10.386981 (+ 22us) write_op.cc:276] Timestamp: P: 1761727922969555 usec, L: 0
1029 08:52:10.387115 (+ 134us) write_op.cc:270] Start()
1029 08:52:10.387139 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922972374 usec, L: 0
1029 08:52:10.387256 (+ 117us) write_op.cc:270] Start()
1029 08:52:10.387280 (+ 24us) write_op.cc:276] Timestamp: P: 1761727922985609 usec, L: 0
1029 08:52:10.387418 (+ 138us) write_op.cc:270] Start()
1029 08:52:10.387437 (+ 19us) write_op.cc:276] Timestamp: P: 1761727922994376 usec, L: 0
1029 08:52:10.387575 (+ 138us) write_op.cc:270] Start()
1029 08:52:10.387593 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923072993 usec, L: 0
1029 08:52:10.387811 (+ 218us) write_op.cc:270] Start()
1029 08:52:10.387834 (+ 23us) write_op.cc:276] Timestamp: P: 1761727923077394 usec, L: 0
1029 08:52:10.387972 (+ 138us) write_op.cc:270] Start()
1029 08:52:10.387990 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923081187 usec, L: 0
1029 08:52:10.388301 (+ 311us) write_op.cc:270] Start()
1029 08:52:10.388339 (+ 38us) write_op.cc:276] Timestamp: P: 1761727923085600 usec, L: 0
1029 08:52:10.388492 (+ 153us) write_op.cc:270] Start()
1029 08:52:10.388513 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923088904 usec, L: 0
1029 08:52:10.388658 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.388679 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923090003 usec, L: 0
1029 08:52:10.388814 (+ 135us) write_op.cc:270] Start()
1029 08:52:10.388839 (+ 25us) write_op.cc:276] Timestamp: P: 1761727923131930 usec, L: 0
1029 08:52:10.389024 (+ 185us) write_op.cc:270] Start()
1029 08:52:10.389050 (+ 26us) write_op.cc:276] Timestamp: P: 1761727923148745 usec, L: 0
1029 08:52:10.389206 (+ 156us) write_op.cc:270] Start()
1029 08:52:10.389227 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923150661 usec, L: 0
1029 08:52:10.389382 (+ 155us) write_op.cc:270] Start()
1029 08:52:10.389439 (+ 57us) write_op.cc:276] Timestamp: P: 1761727923151681 usec, L: 0
1029 08:52:10.389609 (+ 170us) write_op.cc:270] Start()
1029 08:52:10.389643 (+ 34us) write_op.cc:276] Timestamp: P: 1761727923185378 usec, L: 0
1029 08:52:10.389798 (+ 155us) write_op.cc:270] Start()
1029 08:52:10.389822 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923186095 usec, L: 0
1029 08:52:10.389972 (+ 150us) write_op.cc:270] Start()
1029 08:52:10.389993 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923216291 usec, L: 0
1029 08:52:10.390147 (+ 154us) write_op.cc:270] Start()
1029 08:52:10.390168 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923238230 usec, L: 0
1029 08:52:10.390304 (+ 136us) write_op.cc:270] Start()
1029 08:52:10.390338 (+ 34us) write_op.cc:276] Timestamp: P: 1761727923254940 usec, L: 0
1029 08:52:10.390495 (+ 157us) write_op.cc:270] Start()
1029 08:52:10.390518 (+ 23us) write_op.cc:276] Timestamp: P: 1761727923261696 usec, L: 0
1029 08:52:10.390676 (+ 158us) write_op.cc:270] Start()
1029 08:52:10.390696 (+ 20us) write_op.cc:276] Timestamp: P: 1761727923262302 usec, L: 0
1029 08:52:10.390850 (+ 154us) write_op.cc:270] Start()
1029 08:52:10.390869 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923262840 usec, L: 0
1029 08:52:10.390987 (+ 118us) write_op.cc:270] Start()
1029 08:52:10.391004 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923263446 usec, L: 0
1029 08:52:10.391127 (+ 123us) write_op.cc:270] Start()
1029 08:52:10.391146 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923268695 usec, L: 0
1029 08:52:10.391253 (+ 107us) write_op.cc:270] Start()
1029 08:52:10.391269 (+ 16us) write_op.cc:276] Timestamp: P: 1761727923269294 usec, L: 0
1029 08:52:10.391397 (+ 128us) write_op.cc:270] Start()
1029 08:52:10.391415 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923278533 usec, L: 0
1029 08:52:10.391525 (+ 110us) write_op.cc:270] Start()
1029 08:52:10.391544 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923279107 usec, L: 0
1029 08:52:10.391682 (+ 138us) write_op.cc:270] Start()
1029 08:52:10.391700 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923289101 usec, L: 0
1029 08:52:10.391826 (+ 126us) write_op.cc:270] Start()
1029 08:52:10.391843 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923289737 usec, L: 0
1029 08:52:10.391946 (+ 103us) write_op.cc:270] Start()
1029 08:52:10.391962 (+ 16us) write_op.cc:276] Timestamp: P: 1761727923290756 usec, L: 0
1029 08:52:10.392172 (+ 210us) write_op.cc:270] Start()
1029 08:52:10.392195 (+ 23us) write_op.cc:276] Timestamp: P: 1761727923292516 usec, L: 0
1029 08:52:10.392343 (+ 148us) write_op.cc:270] Start()
1029 08:52:10.392365 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923306797 usec, L: 0
1029 08:52:10.392485 (+ 120us) write_op.cc:270] Start()
1029 08:52:10.392502 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923307381 usec, L: 0
1029 08:52:10.392612 (+ 110us) write_op.cc:270] Start()
1029 08:52:10.392630 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923328628 usec, L: 0
1029 08:52:10.392729 (+ 99us) write_op.cc:270] Start()
1029 08:52:10.392745 (+ 16us) write_op.cc:276] Timestamp: P: 1761727923329932 usec, L: 0
1029 08:52:10.392864 (+ 119us) write_op.cc:270] Start()
1029 08:52:10.392880 (+ 16us) write_op.cc:276] Timestamp: P: 1761727923350769 usec, L: 0
1029 08:52:10.393016 (+ 136us) write_op.cc:270] Start()
1029 08:52:10.393033 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923352954 usec, L: 0
1029 08:52:10.393162 (+ 129us) write_op.cc:270] Start()
1029 08:52:10.393181 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923354077 usec, L: 0
1029 08:52:10.393368 (+ 187us) write_op.cc:270] Start()
1029 08:52:10.393392 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923402861 usec, L: 0
1029 08:52:10.393525 (+ 133us) write_op.cc:270] Start()
1029 08:52:10.393543 (+ 18us) write_op.cc:276] Timestamp: P: 1761727923404723 usec, L: 0
1029 08:52:10.393698 (+ 155us) write_op.cc:270] Start()
1029 08:52:10.393720 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923405359 usec, L: 0
1029 08:52:10.393853 (+ 133us) write_op.cc:270] Start()
1029 08:52:10.393872 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923405990 usec, L: 0
1029 08:52:10.408750 (+ 14878us) write_op.cc:270] Start()
1029 08:52:10.408807 (+ 57us) write_op.cc:276] Timestamp: P: 1761727923428392 usec, L: 0
1029 08:52:10.409085 (+ 278us) write_op.cc:270] Start()
1029 08:52:10.409113 (+ 28us) write_op.cc:276] Timestamp: P: 1761727923466241 usec, L: 0
1029 08:52:10.409265 (+ 152us) write_op.cc:270] Start()
1029 08:52:10.409289 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923485201 usec, L: 0
1029 08:52:10.409441 (+ 152us) write_op.cc:270] Start()
1029 08:52:10.409465 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923513849 usec, L: 0
1029 08:52:10.409647 (+ 182us) write_op.cc:270] Start()
1029 08:52:10.409669 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923576218 usec, L: 0
1029 08:52:10.419302 (+ 9633us) write_op.cc:270] Start()
1029 08:52:10.419362 (+ 60us) write_op.cc:276] Timestamp: P: 1761727923592439 usec, L: 0
1029 08:52:10.419879 (+ 517us) write_op.cc:270] Start()
1029 08:52:10.419908 (+ 29us) write_op.cc:276] Timestamp: P: 1761727923618702 usec, L: 0
1029 08:52:10.420100 (+ 192us) write_op.cc:270] Start()
1029 08:52:10.420122 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923636810 usec, L: 0
1029 08:52:10.420274 (+ 152us) write_op.cc:270] Start()
1029 08:52:10.420301 (+ 27us) write_op.cc:276] Timestamp: P: 1761727923661722 usec, L: 0
1029 08:52:10.420469 (+ 168us) write_op.cc:270] Start()
1029 08:52:10.420489 (+ 20us) write_op.cc:276] Timestamp: P: 1761727923674635 usec, L: 0
1029 08:52:10.420641 (+ 152us) write_op.cc:270] Start()
1029 08:52:10.672844 (+252203us) write_op.cc:276] Timestamp: P: 1761727923689341 usec, L: 0
1029 08:52:10.673106 (+ 262us) write_op.cc:270] Start()
1029 08:52:10.673144 (+ 38us) write_op.cc:276] Timestamp: P: 1761727923704338 usec, L: 0
1029 08:52:10.673353 (+ 209us) write_op.cc:270] Start()
1029 08:52:10.673375 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923721307 usec, L: 0
1029 08:52:10.673523 (+ 148us) write_op.cc:270] Start()
1029 08:52:10.673547 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923731227 usec, L: 0
1029 08:52:10.673680 (+ 133us) write_op.cc:270] Start()
1029 08:52:10.673697 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923744644 usec, L: 0
1029 08:52:10.673821 (+ 124us) write_op.cc:270] Start()
1029 08:52:10.673840 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923795717 usec, L: 0
1029 08:52:10.673974 (+ 134us) write_op.cc:270] Start()
1029 08:52:10.673993 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923824557 usec, L: 0
1029 08:52:10.674139 (+ 146us) write_op.cc:270] Start()
1029 08:52:10.674158 (+ 19us) write_op.cc:276] Timestamp: P: 1761727923857359 usec, L: 0
1029 08:52:10.674355 (+ 197us) write_op.cc:270] Start()
1029 08:52:10.674377 (+ 22us) write_op.cc:276] Timestamp: P: 1761727923873654 usec, L: 0
1029 08:52:10.674510 (+ 133us) write_op.cc:270] Start()
1029 08:52:10.674530 (+ 20us) write_op.cc:276] Timestamp: P: 1761727923906765 usec, L: 0
1029 08:52:10.674687 (+ 157us) write_op.cc:270] Start()
1029 08:52:10.674708 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923930109 usec, L: 0
1029 08:52:10.674867 (+ 159us) write_op.cc:270] Start()
1029 08:52:10.674891 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923932576 usec, L: 0
1029 08:52:10.675023 (+ 132us) write_op.cc:270] Start()
1029 08:52:10.675047 (+ 24us) write_op.cc:276] Timestamp: P: 1761727923933441 usec, L: 0
1029 08:52:10.675181 (+ 134us) write_op.cc:270] Start()
1029 08:52:10.675223 (+ 42us) write_op.cc:276] Timestamp: P: 1761727923939662 usec, L: 0
1029 08:52:10.675373 (+ 150us) write_op.cc:270] Start()
1029 08:52:10.675396 (+ 23us) write_op.cc:276] Timestamp: P: 1761727923958098 usec, L: 0
1029 08:52:10.675546 (+ 150us) write_op.cc:270] Start()
1029 08:52:10.675569 (+ 23us) write_op.cc:276] Timestamp: P: 1761727923960887 usec, L: 0
1029 08:52:10.675732 (+ 163us) write_op.cc:270] Start()
1029 08:52:10.675749 (+ 17us) write_op.cc:276] Timestamp: P: 1761727923961715 usec, L: 0
1029 08:52:10.675897 (+ 148us) write_op.cc:270] Start()
1029 08:52:10.675918 (+ 21us) write_op.cc:276] Timestamp: P: 1761727923970297 usec, L: 0
1029 08:52:10.740425 (+ 64507us) write_op.cc:270] Start()
1029 08:52:10.740473 (+ 48us) write_op.cc:276] Timestamp: P: 1761727923972181 usec, L: 0
1029 08:52:10.740753 (+ 280us) write_op.cc:270] Start()
1029 08:52:10.740784 (+ 31us) write_op.cc:276] Timestamp: P: 1761727923980770 usec, L: 0
1029 08:52:10.740944 (+ 160us) write_op.cc:270] Start()
1029 08:52:10.740966 (+ 22us) write_op.cc:276] Timestamp: P: 1761727924012249 usec, L: 0
1029 08:52:10.741106 (+ 140us) write_op.cc:270] Start()
1029 08:52:10.741129 (+ 23us) write_op.cc:276] Timestamp: P: 1761727924035123 usec, L: 0
1029 08:52:10.741278 (+ 149us) write_op.cc:270] Start()
1029 08:52:10.797572 (+ 56294us) write_op.cc:276] Timestamp: P: 1761727924035716 usec, L: 0
1029 08:52:10.797887 (+ 315us) write_op.cc:270] Start()
1029 08:52:10.797928 (+ 41us) write_op.cc:276] Timestamp: P: 1761727924172254 usec, L: 0
1029 08:52:10.798088 (+ 160us) write_op.cc:270] Start()
1029 08:52:10.798109 (+ 21us) write_op.cc:276] Timestamp: P: 1761727924173518 usec, L: 0
1029 08:52:10.798225 (+ 116us) write_op.cc:270] Start()
1029 08:52:10.798255 (+ 30us) write_op.cc:276] Timestamp: P: 1761727924174178 usec, L: 0
1029 08:52:10.798418 (+ 163us) write_op.cc:270] Start()
1029 08:52:10.798444 (+ 26us) write_op.cc:276] Timestamp: P: 1761727924174814 usec, L: 0
1029 08:52:10.798593 (+ 149us) write_op.cc:270] Start()
1029 08:52:10.798615 (+ 22us) write_op.cc:276] Timestamp: P: 1761727924187547 usec, L: 0
1029 08:52:10.798785 (+ 170us) write_op.cc:270] Start()
1029 08:52:10.798810 (+ 25us) write_op.cc:276] Timestamp: P: 1761727924197615 usec, L: 0
1029 08:52:10.798970 (+ 160us) write_op.cc:270] Start()
1029 08:52:10.798989 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924201284 usec, L: 0
1029 08:52:10.799134 (+ 145us) write_op.cc:270] Start()
1029 08:52:10.799155 (+ 21us) write_op.cc:276] Timestamp: P: 1761727924203044 usec, L: 0
1029 08:52:10.799323 (+ 168us) write_op.cc:270] Start()
1029 08:52:10.799342 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924203664 usec, L: 0
1029 08:52:10.799495 (+ 153us) write_op.cc:270] Start()
1029 08:52:10.799518 (+ 23us) write_op.cc:276] Timestamp: P: 1761727924209547 usec, L: 0
1029 08:52:10.799671 (+ 153us) write_op.cc:270] Start()
1029 08:52:10.799694 (+ 23us) write_op.cc:276] Timestamp: P: 1761727924231476 usec, L: 0
1029 08:52:10.799834 (+ 140us) write_op.cc:270] Start()
1029 08:52:10.799854 (+ 20us) write_op.cc:276] Timestamp: P: 1761727924235037 usec, L: 0
1029 08:52:10.811629 (+ 11775us) write_op.cc:270] Start()
1029 08:52:10.813163 (+ 1534us) write_op.cc:276] Timestamp: P: 1761727924244811 usec, L: 0
1029 08:52:10.820357 (+ 7194us) write_op.cc:270] Start()
1029 08:52:10.821240 (+ 883us) write_op.cc:276] Timestamp: P: 1761727924247432 usec, L: 0
1029 08:52:10.827364 (+ 6124us) write_op.cc:270] Start()
1029 08:52:10.827700 (+ 336us) write_op.cc:276] Timestamp: P: 1761727924248043 usec, L: 0
1029 08:52:10.832774 (+ 5074us) write_op.cc:270] Start()
1029 08:52:10.833095 (+ 321us) write_op.cc:276] Timestamp: P: 1761727924250529 usec, L: 0
1029 08:52:10.837153 (+ 4058us) write_op.cc:270] Start()
1029 08:52:10.837695 (+ 542us) write_op.cc:276] Timestamp: P: 1761727924251244 usec, L: 0
1029 08:52:10.841954 (+ 4259us) write_op.cc:270] Start()
1029 08:52:10.843555 (+ 1601us) write_op.cc:276] Timestamp: P: 1761727924251814 usec, L: 0
1029 08:52:10.849036 (+ 5481us) write_op.cc:270] Start()
1029 08:52:10.850334 (+ 1298us) write_op.cc:276] Timestamp: P: 1761727924274390 usec, L: 0
1029 08:52:10.857401 (+ 7067us) write_op.cc:270] Start()
1029 08:52:10.858365 (+ 964us) write_op.cc:276] Timestamp: P: 1761727924275070 usec, L: 0
1029 08:52:10.863828 (+ 5463us) write_op.cc:270] Start()
1029 08:52:10.864378 (+ 550us) write_op.cc:276] Timestamp: P: 1761727924292288 usec, L: 0
1029 08:52:10.874218 (+ 9840us) write_op.cc:270] Start()
1029 08:52:10.875702 (+ 1484us) write_op.cc:276] Timestamp: P: 1761727924301410 usec, L: 0
1029 08:52:11.177977 (+302275us) write_op.cc:270] Start()
1029 08:52:11.178026 (+ 49us) write_op.cc:276] Timestamp: P: 1761727924311686 usec, L: 0
1029 08:52:11.178226 (+ 200us) write_op.cc:270] Start()
1029 08:52:11.178255 (+ 29us) write_op.cc:276] Timestamp: P: 1761727924356585 usec, L: 0
1029 08:52:11.178398 (+ 143us) write_op.cc:270] Start()
1029 08:52:11.178417 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924388683 usec, L: 0
1029 08:52:11.178557 (+ 140us) write_op.cc:270] Start()
1029 08:52:11.178577 (+ 20us) write_op.cc:276] Timestamp: P: 1761727924436167 usec, L: 0
1029 08:52:11.178717 (+ 140us) write_op.cc:270] Start()
1029 08:52:11.178787 (+ 70us) write_op.cc:276] Timestamp: P: 1761727924440798 usec, L: 0
1029 08:52:11.178958 (+ 171us) write_op.cc:270] Start()
1029 08:52:11.178976 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924467062 usec, L: 0
1029 08:52:11.179098 (+ 122us) write_op.cc:270] Start()
1029 08:52:11.179114 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924510717 usec, L: 0
1029 08:52:11.179225 (+ 111us) write_op.cc:270] Start()
1029 08:52:11.179243 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924515144 usec, L: 0
1029 08:52:11.179392 (+ 149us) write_op.cc:270] Start()
1029 08:52:11.179420 (+ 28us) write_op.cc:276] Timestamp: P: 1761727924515793 usec, L: 0
1029 08:52:11.179576 (+ 156us) write_op.cc:270] Start()
1029 08:52:11.179597 (+ 21us) write_op.cc:276] Timestamp: P: 1761727924567493 usec, L: 0
1029 08:52:11.179728 (+ 131us) write_op.cc:270] Start()
1029 08:52:11.179745 (+ 17us) write_op.cc:276] Timestamp: P: 1761727924602228 usec, L: 0
1029 08:52:11.179856 (+ 111us) write_op.cc:270] Start()
1029 08:52:11.179872 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924608796 usec, L: 0
1029 08:52:11.179972 (+ 100us) write_op.cc:270] Start()
1029 08:52:11.179988 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924609790 usec, L: 0
1029 08:52:11.214267 (+ 34279us) write_op.cc:270] Start()
1029 08:52:11.214525 (+ 258us) write_op.cc:276] Timestamp: P: 1761727924610428 usec, L: 0
1029 08:52:11.217619 (+ 3094us) write_op.cc:270] Start()
1029 08:52:11.217755 (+ 136us) write_op.cc:276] Timestamp: P: 1761727924640699 usec, L: 0
1029 08:52:11.218897 (+ 1142us) write_op.cc:270] Start()
1029 08:52:11.219034 (+ 137us) write_op.cc:276] Timestamp: P: 1761727924644730 usec, L: 0
1029 08:52:11.340280 (+121246us) write_op.cc:270] Start()
1029 08:52:11.340330 (+ 50us) write_op.cc:276] Timestamp: P: 1761727924646765 usec, L: 0
1029 08:52:11.340546 (+ 216us) write_op.cc:270] Start()
1029 08:52:11.340584 (+ 38us) write_op.cc:276] Timestamp: P: 1761727924661153 usec, L: 0
1029 08:52:11.340734 (+ 150us) write_op.cc:270] Start()
1029 08:52:11.340760 (+ 26us) write_op.cc:276] Timestamp: P: 1761727924676871 usec, L: 0
1029 08:52:11.341017 (+ 257us) write_op.cc:270] Start()
1029 08:52:11.341037 (+ 20us) write_op.cc:276] Timestamp: P: 1761727924691894 usec, L: 0
1029 08:52:11.341166 (+ 129us) write_op.cc:270] Start()
1029 08:52:11.341183 (+ 17us) write_op.cc:276] Timestamp: P: 1761727924712382 usec, L: 0
1029 08:52:11.341292 (+ 109us) write_op.cc:270] Start()
1029 08:52:11.341310 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924725314 usec, L: 0
1029 08:52:11.341435 (+ 125us) write_op.cc:270] Start()
1029 08:52:11.341454 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924736895 usec, L: 0
1029 08:52:11.341572 (+ 118us) write_op.cc:270] Start()
1029 08:52:11.341591 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924738967 usec, L: 0
1029 08:52:11.341726 (+ 135us) write_op.cc:270] Start()
1029 08:52:11.341744 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924776899 usec, L: 0
1029 08:52:11.341866 (+ 122us) write_op.cc:270] Start()
1029 08:52:11.341884 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924789689 usec, L: 0
1029 08:52:11.342004 (+ 120us) write_op.cc:270] Start()
1029 08:52:11.342020 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924801853 usec, L: 0
1029 08:52:11.342164 (+ 144us) write_op.cc:270] Start()
1029 08:52:11.342182 (+ 18us) write_op.cc:276] Timestamp: P: 1761727924816191 usec, L: 0
1029 08:52:11.342323 (+ 141us) write_op.cc:270] Start()
1029 08:52:11.342342 (+ 19us) write_op.cc:276] Timestamp: P: 1761727924821741 usec, L: 0
1029 08:52:11.342476 (+ 134us) write_op.cc:270] Start()
1029 08:52:11.342493 (+ 17us) write_op.cc:276] Timestamp: P: 1761727924852668 usec, L: 0
1029 08:52:11.342620 (+ 127us) write_op.cc:270] Start()
1029 08:52:11.342637 (+ 17us) write_op.cc:276] Timestamp: P: 1761727924891471 usec, L: 0
1029 08:52:11.342736 (+ 99us) write_op.cc:270] Start()
1029 08:52:11.342752 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924892189 usec, L: 0
1029 08:52:11.342866 (+ 114us) write_op.cc:270] Start()
1029 08:52:11.342881 (+ 15us) write_op.cc:276] Timestamp: P: 1761727924906223 usec, L: 0
1029 08:52:11.342983 (+ 102us) write_op.cc:270] Start()
1029 08:52:11.342999 (+ 16us) write_op.cc:276] Timestamp: P: 1761727924921493 usec, L: 0
1029 08:52:11.343129 (+ 130us) write_op.cc:270] Start()
1029 08:52:11.343152 (+ 23us) write_op.cc:276] Timestamp: P: 17617279
I20251029 08:52:12.532908 1040 tablet_service.cc:1505] Processing CreateTablet for tablet b5e1e3ce83b14817b2440370cb2fcabd (DEFAULT_TABLE table=default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.533346 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 0c14c63f74f74a18902a17a4d210a90d (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.533752 1179 raft_consensus.cc:2272] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.534819 1687 raft_consensus.cc:493] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.535036 1687 raft_consensus.cc:515] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.535919 1687 leader_election.cc:290] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.524739 1715 leader_election.cc:290] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:12.537194 1176 tablet_replica.cc:333] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.537571 1176 raft_consensus.cc:2243] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.539420 1176 raft_consensus.cc:2272] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.540172 1176 ts_tablet_manager.cc:1916] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.545267 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bf5d693be240f18bacccc81a6ea5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.551877 1179 ts_tablet_manager.cc:1916] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.552623 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca466714c774fc9982ef145c6dd1571 (DEFAULT_TABLE table=default.Table_ea2ca0e66f9c40d1971ec171d61737fb [id=0a9b61d3c85941df84deaf0250a2a557]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.553153 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca466714c774fc9982ef145c6dd1571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.556259 1751 raft_consensus.cc:697] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:12.556645 1751 consensus_queue.cc:237] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:12.556232 996 leader_election.cc:343] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a2947083e3e447d5addae5effd186eb8
I20251029 08:52:12.557821 1183 ts_tablet_manager.cc:1434] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:12.558224 1183 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:12.562991 996 leader_election.cc:304] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:12.563627 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e1e3ce83b14817b2440370cb2fcabd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.564673 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 46553.6us
I20251029 08:52:12.565681 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dee931aa035748d08b1260e8ab3fca5c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.566059 1176 ts_tablet_manager.cc:1929] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.566216 1176 log.cc:1199] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8626877e3cfc4cb48f6caa92638b4917
W20251029 08:52:12.567463 720 leader_election.cc:343] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.571063 1757 raft_consensus.cc:2749] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.575640 1176 ts_tablet_manager.cc:1950] T 8626877e3cfc4cb48f6caa92638b4917 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.575966 1183 tablet_bootstrap.cc:654] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.576954 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8626877e3cfc4cb48f6caa92638b4917 (table default.Table_c43c6468d7c447b28f338ce82a1df720 [id=38e7e568f136480caaa0ee51ea6927f3]) successfully deleted
W20251029 08:52:12.581457 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a2b7853b85d24a04b8b11caaa4fe8c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2b7853b85d24a04b8b11caaa4fe8c4d already in progress: creating tablet
I20251029 08:52:12.582508 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14c63f74f74a18902a17a4d210a90d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.583483 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c7024e6eb4124109ad1575d216739fe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7024e6eb4124109ad1575d216739fe2 already in progress: creating tablet
W20251029 08:52:12.584251 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2ca466714c774fc9982ef145c6dd1571 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca466714c774fc9982ef145c6dd1571 already in progress: creating tablet
W20251029 08:52:12.585927 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 1dd62833c9d548adabbbf71856a4b181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dd62833c9d548adabbbf71856a4b181 already in progress: creating tablet
I20251029 08:52:12.586288 1176 tablet_replica.cc:333] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.587446 1176 raft_consensus.cc:2243] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.588927 1176 raft_consensus.cc:2272] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.589910 1176 ts_tablet_manager.cc:1916] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.617830 1177 raft_consensus.cc:740] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.630609 1175 ts_tablet_manager.cc:1929] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:12.630852 1175 log.cc:1199] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c8cb0c6c6a1c4877a89476d4822f48db
I20251029 08:52:12.631419 1175 ts_tablet_manager.cc:1950] T c8cb0c6c6a1c4877a89476d4822f48db P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:12.638093 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e1e3ce83b14817b2440370cb2fcabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.638562 1749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:12.638928 1749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:12.642755 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 364) took 3399 ms. Trace:
I20251029 08:52:12.642540 1177 consensus_queue.cc:260] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.646631 1176 ts_tablet_manager.cc:1929] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.646808 1176 log.cc:1199] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/dee931aa035748d08b1260e8ab3fca5c
I20251029 08:52:12.647183 1176 ts_tablet_manager.cc:1950] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.648012 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet dee931aa035748d08b1260e8ab3fca5c (table default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]) successfully deleted
I20251029 08:52:12.650172 1175 rpcz_store.cc:276] 1029 08:52:09.233049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.233952 (+ 903us) service_pool.cc:225] Handling call
1029 08:52:12.190274 (+2956322us) ts_tablet_manager.cc:1151] Deleting tablet c8cb0c6c6a1c4877a89476d4822f48db
1029 08:52:12.190289 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.286748 (+ 96459us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.615737 (+328989us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.632353 (+ 16616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2951033}
W20251029 08:52:12.653715 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e5fa181849474a559701fdb8c18d398b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5fa181849474a559701fdb8c18d398b already in progress: creating tablet
W20251029 08:52:12.656037 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 072cf27b0743488c8166e03b5e61c545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072cf27b0743488c8166e03b5e61c545 already in progress: creating tablet
I20251029 08:52:12.656523 1176 tablet_replica.cc:333] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.658177 1176 raft_consensus.cc:2243] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.658504 1176 raft_consensus.cc:2272] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.665529 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_21761ed98ca84ce58dcd72a3dac157c1" } new_table_name: "default.Table_bdbb78a8f0ef452498a4b45c1e29b0c7" modify_external_catalogs: true
I20251029 08:52:12.671559 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 361) took 3430 ms. Trace:
I20251029 08:52:12.686985 1176 ts_tablet_manager.cc:1916] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:12.688596 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bd490da0b0e47fc965343c123a27b22 consensus state has no leader
I20251029 08:52:12.691771 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ebfdc32c8a6f48359f16be99eaef1443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.692349 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebfdc32c8a6f48359f16be99eaef1443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.695907 1773 raft_consensus.cc:2749] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.696542 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7f4559dca42a45bd9d4f2e050a759faa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.697110 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f4559dca42a45bd9d4f2e050a759faa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.702379 1741 raft_consensus.cc:697] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:12.700700 1175 rpcz_store.cc:276] 1029 08:52:09.232696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.233024 (+ 328us) service_pool.cc:225] Handling call
1029 08:52:12.650627 (+3417603us) ts_tablet_manager.cc:1151] Deleting tablet 215cd666c71a4179a19e298868a88e96
1029 08:52:12.650639 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.662890 (+ 12251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3392991}
I20251029 08:52:12.711663 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c8cb0c6c6a1c4877a89476d4822f48db (table default.Table_42cce27285944566a9037e99c90de248 [id=c99d60e34d3a4ba1b1a36319e7584543]) successfully deleted
I20251029 08:52:12.712231 1179 ts_tablet_manager.cc:1929] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:12.712477 1179 log.cc:1199] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/57e2b19a227740a28ae601d77e62597b
I20251029 08:52:12.703290 1741 consensus_queue.cc:237] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.703847 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 363) took 3470 ms. Trace:
I20251029 08:52:12.701170 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dee931aa035748d08b1260e8ab3fca5c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:12.718276 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 215cd666c71a4179a19e298868a88e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215cd666c71a4179a19e298868a88e96 already in progress: creating tablet
I20251029 08:52:12.719161 1175 rpcz_store.cc:276] 1029 08:52:09.232945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.233883 (+ 938us) service_pool.cc:225] Handling call
1029 08:52:12.703687 (+3469804us) ts_tablet_manager.cc:1151] Deleting tablet bd0c43367a8c47f390cc61d2764d61e4
1029 08:52:12.703698 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.703806 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3422956}
I20251029 08:52:12.719521 1175 tablet_replica.cc:333] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:12.721913 1175 raft_consensus.cc:2243] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.722112 1175 raft_consensus.cc:2272] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.723248 1175 ts_tablet_manager.cc:1916] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.724975 1773 raft_consensus.cc:493] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:12.726540 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85b300f17a704f7dba1aa3724fcc2de7 consensus state has no leader
I20251029 08:52:12.727032 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4f60439c93314f89909e3032858f1dab
I20251029 08:52:12.729022 1179 ts_tablet_manager.cc:1950] T 57e2b19a227740a28ae601d77e62597b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:12.729077 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_08e6f2532b724e02acbd01a2d18fb7a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.729727 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08e6f2532b724e02acbd01a2d18fb7a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.731024 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_685d1d31eb774418a8e997595f7e8ff8" } new_table_name: "default.Table_e4e823c7272f44e1af831ef60277d505" modify_external_catalogs: true
W20251029 08:52:12.733291 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 225626us
W20251029 08:52:12.733534 861 leader_election.cc:343] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.733690 861 leader_election.cc:304] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:12.733741 1178 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:12.734016 1178 ts_tablet_manager.cc:1403] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.396s user 0.114s sys 0.173s
W20251029 08:52:12.734088 861 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.734220 861 leader_election.cc:304] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:12.734566 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 467) took 1554 ms. Trace:
I20251029 08:52:12.734812 1179 rpcz_store.cc:276] 1029 08:52:11.180449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.180515 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:11.916469 (+735954us) ts_tablet_manager.cc:1151] Deleting tablet 57e2b19a227740a28ae601d77e62597b
1029 08:52:11.916476 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.359772 (+443296us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x6160000f3454. stack: 00007f0aa8bae9f6 00007f0aa8bae7b0 00007f0aa7b8a3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f0ab4a79100 00007f0ab5dc6ae3 00007f0ab8987287 00007f0ab89ba2d4 00007f0ab899ad70 00007f0ab899aafd 0000000000a8867c 00007f0aa8bf4d7f 00007f0aa8bf804b 00007f0aa8bf7ec8
1029 08:52:12.497515 (+137743us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x6170000745f0. stack: 00007f0aa8bae9f6 00007f0aa8bae7b0 00007f0aa7b8a3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f0ab495876f 00007f0ab4a08f5c 00007f0ab4a791e3 00007f0ab5dc6ae3 00007f0ab8987287 00007f0ab89ba2d4 00007f0ab899ad70 00007f0ab899aafd 0000000000a8867c 00007f0aa8bf4d7f
1029 08:52:12.513361 (+ 15846us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x617000074270. stack: 00007f0aa8bae9f6 00007f0aa8bae7b0 00007f0aa7b8a3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f0ab495876f 00007f0ab4a08f5c 00007f0ab4a791e3 00007f0ab5dc6ae3 00007f0ab8987287 00007f0ab89ba2d4 00007f0ab899ad70 00007f0ab899aafd 0000000000a8867c 00007f0aa8bf4d7f
1029 08:52:12.581292 (+ 67931us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.710728 (+129436us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.734525 (+ 23797us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203,"spinlock_wait_cycles":1057067392,"tablet-delete.queue_time_us":735625}
I20251029 08:52:12.736490 1179 tablet_replica.cc:333] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:12.736433 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 57e2b19a227740a28ae601d77e62597b (table default.Table_ab2c1cce4615467595cb78eb6ca1b4ec [id=d283eaf060754468a34dfd1849f618ad]) successfully deleted
I20251029 08:52:12.740586 1179 raft_consensus.cc:2243] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.740401 1773 raft_consensus.cc:515] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.743765 1773 leader_election.cc:290] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:12.747104 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57fcb0952cf24438aadd4476f7f120a7" } modify_external_catalogs: true
I20251029 08:52:12.747396 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57fcb0952cf24438aadd4476f7f120a7" } modify_external_catalogs: true
I20251029 08:52:12.748319 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e9b105d36aed4a0da8586038f56dec12" } modify_external_catalogs: true
I20251029 08:52:12.748530 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e9b105d36aed4a0da8586038f56dec12" } modify_external_catalogs: true
I20251029 08:52:12.749056 593 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b0000aa4bc6d4074995fad9444d6dad2
I20251029 08:52:12.749446 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_95e4062d439d4ac48a856012207ec7f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.750221 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95e4062d439d4ac48a856012207ec7f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.752558 1178 raft_consensus.cc:359] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.752795 1178 raft_consensus.cc:385] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.752912 1178 raft_consensus.cc:740] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.753165 1178 consensus_queue.cc:260] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.753768 1178 ts_tablet_manager.cc:1434] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251029 08:52:12.755515 1178 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:12.757366 1178 tablet_bootstrap.cc:654] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.759828 1176 ts_tablet_manager.cc:1929] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.759958 1176 log.cc:1199] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/74fe974a368a4afe8a5da6dfd1036400
I20251029 08:52:12.760421 1176 ts_tablet_manager.cc:1950] T 74fe974a368a4afe8a5da6dfd1036400 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.763556 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 74fe974a368a4afe8a5da6dfd1036400 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432]) successfully deleted
I20251029 08:52:12.764925 1772 raft_consensus.cc:2749] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:12.766763 718 leader_election.cc:343] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.766916 718 leader_election.cc:304] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:12.767120 718 raft_consensus.cc:2686] T dee931aa035748d08b1260e8ab3fca5c P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:12.768704 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bd0c43367a8c47f390cc61d2764d61e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd0c43367a8c47f390cc61d2764d61e4 already in progress: creating tablet
I20251029 08:52:12.771268 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a9cf61d6137544019bca25acf2431986
I20251029 08:52:12.772127 1773 raft_consensus.cc:2749] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.772428 761 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cf61d6137544019bca25acf2431986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:12.774531 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cf61d6137544019bca25acf2431986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.774650 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:12.776350 1775 tablet.cc:1722] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:12.778136 1779 raft_consensus.cc:493] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:12.781898 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0ff0658e0274cf49c389ad602d91b03 consensus state has no leader
I20251029 08:52:12.782943 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7792ee223dbe43aa82990541caced045" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.783331 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cf61d6137544019bca25acf2431986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.784163 1751 raft_consensus.cc:493] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.784373 1751 raft_consensus.cc:515] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.784834 1779 raft_consensus.cc:515] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:12.786242 998 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:12.786423 1179 raft_consensus.cc:2272] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.787436 1179 ts_tablet_manager.cc:1916] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.788954 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09759a50a82f4b758449da9fbbc1f72d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:12.790244 1183 tablet_bootstrap.cc:492] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:12.791882 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2fc69b9cc1354fc1aba2e2bccac2e095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.792923 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fc69b9cc1354fc1aba2e2bccac2e095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.794502 1751 leader_election.cc:290] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.794606 1779 leader_election.cc:290] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:12.796527 1183 ts_tablet_manager.cc:1403] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.237s user 0.008s sys 0.006s
I20251029 08:52:12.797441 1183 raft_consensus.cc:359] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.797752 1183 raft_consensus.cc:385] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.797905 1183 raft_consensus.cc:740] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.798261 1183 consensus_queue.cc:260] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.798872 1183 ts_tablet_manager.cc:1434] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:12.798940 1176 tablet_replica.cc:333] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:12.799189 1176 raft_consensus.cc:2243] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.799413 1176 raft_consensus.cc:2272] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.799428 1183 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:12.800105 1176 ts_tablet_manager.cc:1916] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.800305 922 raft_consensus.cc:2468] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:12.801800 996 leader_election.cc:304] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:12.802449 1751 raft_consensus.cc:2804] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:12.802680 1751 raft_consensus.cc:493] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:12.802886 1751 raft_consensus.cc:3060] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.807538 1751 raft_consensus.cc:515] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:12.808374 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f01d88c516e14e62b7f491656c59b75a has no consensus state
I20251029 08:52:12.808831 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c6c5865ac9e944ea97b59f1b9e274284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.809399 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6c5865ac9e944ea97b59f1b9e274284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.822615 1175 ts_tablet_manager.cc:1929] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.828625 1175 log.cc:1199] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4cf906c73dc84403bcc3dece74a822bd
I20251029 08:52:12.829110 1175 ts_tablet_manager.cc:1950] T 4cf906c73dc84403bcc3dece74a822bd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:12.831045 596 master_service.cc:954] ReplaceTablet: received request to replace tablet c572c68b32d443b2a019e1f4bf2d82fe from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:12.837100 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4deeec492a2545ca86583c6ea899a0ab" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:12.831477 861 consensus_peers.cc:597] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aea79bd4e6844b7abac623824cabbeeb. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:12.837718 1177 ts_tablet_manager.cc:1434] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.424s user 0.003s sys 0.012s
I20251029 08:52:12.840616 1177 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:12.843176 1176 ts_tablet_manager.cc:1929] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:12.843348 1176 log.cc:1199] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a9cf61d6137544019bca25acf2431986
I20251029 08:52:12.843803 1176 ts_tablet_manager.cc:1950] T a9cf61d6137544019bca25acf2431986 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:12.844882 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]) successfully deleted
W20251029 08:52:12.848068 861 leader_election.cc:343] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7792ee223dbe43aa82990541caced045
W20251029 08:52:12.848408 861 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:12.849483 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 375) took 3413 ms. Trace:
I20251029 08:52:12.850015 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "983a406aeed243f29ab322f94a0f5239" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:12.853209 1183 tablet_bootstrap.cc:654] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:12.853444 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a9f0b579ebd4d2db813459611d7f9bf consensus state has no leader
I20251029 08:52:12.849632 1175 rpcz_store.cc:276] 1029 08:52:09.416763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.418681 (+ 1918us) service_pool.cc:225] Handling call
1029 08:52:12.719452 (+3300771us) ts_tablet_manager.cc:1151] Deleting tablet 4cf906c73dc84403bcc3dece74a822bd
1029 08:52:12.719464 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.776127 (+ 56663us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.822480 (+ 46353us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.830204 (+ 7724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3300507}
I20251029 08:52:12.856509 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4cf906c73dc84403bcc3dece74a822bd (table default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]) successfully deleted
W20251029 08:52:12.849728 861 leader_election.cc:343] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.856889 861 leader_election.cc:304] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:12.857383 861 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.857499 861 leader_election.cc:304] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:12.858713 1779 raft_consensus.cc:2749] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.858913 1779 raft_consensus.cc:2749] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:12.859418 861 leader_election.cc:343] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:12.859612 861 leader_election.cc:343] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 983a406aeed243f29ab322f94a0f5239
I20251029 08:52:12.859715 861 leader_election.cc:304] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:12.860123 861 consensus_peers.cc:597] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9cf61d6137544019bca25acf2431986. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:12.860610 1781 raft_consensus.cc:2749] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:12.853070 1177 tablet_bootstrap.cc:654] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.864495 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 377) took 3447 ms. Trace:
I20251029 08:52:12.864609 1175 rpcz_store.cc:276] 1029 08:52:09.417041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.420508 (+ 3467us) service_pool.cc:225] Handling call
1029 08:52:12.864299 (+3443791us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:12.864311 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.864445 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3443245}
W20251029 08:52:12.865042 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c66cbb495f5142d79a674f7d5adc505e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c66cbb495f5142d79a674f7d5adc505e already in progress: creating tablet
I20251029 08:52:12.865393 1175 tablet_replica.cc:333] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:12.865595 1175 raft_consensus.cc:2243] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.865779 1175 raft_consensus.cc:2272] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.866405 1175 ts_tablet_manager.cc:1916] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.867862 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 57e2b19a227740a28ae601d77e62597b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.870152 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_acc324f38e8943f4995a5f0bf3be0280"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.870781 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc324f38e8943f4995a5f0bf3be0280 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.871400 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09759a50a82f4b758449da9fbbc1f72d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:12.871644 922 raft_consensus.cc:3060] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:12.872697 1751 leader_election.cc:290] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.874671 1179 ts_tablet_manager.cc:1929] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.874846 1179 log.cc:1199] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a7fdbe6681894ffa8264cdfa486f16e6
I20251029 08:52:12.875039 922 raft_consensus.cc:2468] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:12.875579 996 leader_election.cc:304] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:12.876281 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet a57b1dbbdf854bc6a6e1b5dc88b63bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:12.876992 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 09759a50a82f4b758449da9fbbc1f72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:12.877247 998 leader_election.cc:343] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:12.878230 1751 raft_consensus.cc:2804] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:12.878305 1784 tablet.cc:1722] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:12.879870 1751 raft_consensus.cc:697] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:12.884699 1775 tablet.cc:1717] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:12.883744 1751 consensus_queue.cc:237] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.891492 1177 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:12.892236 1177 ts_tablet_manager.cc:1403] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.003s
I20251029 08:52:12.893040 1177 raft_consensus.cc:359] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.893280 1177 raft_consensus.cc:385] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.893368 1177 raft_consensus.cc:740] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.893604 1177 consensus_queue.cc:260] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:12.895798 1177 ts_tablet_manager.cc:1434] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:12.896129 1177 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:12.897691 1179 ts_tablet_manager.cc:1950] T a7fdbe6681894ffa8264cdfa486f16e6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:12.904883 1177 tablet_bootstrap.cc:654] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:12.906432 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9e95376692ff46ceb83308b096d8cd24" } new_table_name: "default.Table_fd8c4e0f158b4582923e9de97aa89bcf" modify_external_catalogs: true
I20251029 08:52:12.910465 1178 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:12.912189 1178 ts_tablet_manager.cc:1403] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.157s user 0.001s sys 0.011s
I20251029 08:52:12.913344 1175 ts_tablet_manager.cc:1929] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:12.913483 1175 log.cc:1199] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:12.913574 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:12.913880 1175 ts_tablet_manager.cc:1950] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:12.914731 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 384) took 3451 ms. Trace:
I20251029 08:52:12.914847 1175 rpcz_store.cc:276] 1029 08:52:09.463149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.463406 (+ 257us) service_pool.cc:225] Handling call
1029 08:52:12.865311 (+3401905us) ts_tablet_manager.cc:1151] Deleting tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f
1029 08:52:12.865322 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.869926 (+ 4604us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.913171 (+ 43245us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.914694 (+ 1523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3350772}
I20251029 08:52:12.916402 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_090993e9452e40a68d67f5a3a00cf3e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:12.917083 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_090993e9452e40a68d67f5a3a00cf3e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.918697 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:12.919548 893 tablet_service.cc:1505] Processing CreateTablet for tablet f7b89ced286b4a04a28022917b1d0235 (DEFAULT_TABLE table=default.Table_a3fb78e6a50342688581081eb505f8ee [id=215977502aab462a803071559b8021a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.919726 888 tablet_service.cc:1505] Processing CreateTablet for tablet 8a76f229c6264d4ea65c5b9733465a62 (DEFAULT_TABLE table=default.Table_051b538b21fe4b6b8ea44de581730151 [id=56f690e90bec427f970ff0d8ef7faf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.920007 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b89ced286b4a04a28022917b1d0235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.919929 902 tablet_service.cc:1505] Processing CreateTablet for tablet c8d40479b421435eada7682fd7f745b2 (DEFAULT_TABLE table=default.Table_b37a9d683d684befa22f0c33ef953739 [id=b0d8f7f22e8d44039d8ee280b893dc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.920279 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a76f229c6264d4ea65c5b9733465a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.920480 892 tablet_service.cc:1505] Processing CreateTablet for tablet f01d88c516e14e62b7f491656c59b75a (DEFAULT_TABLE table=default.Table_bdbb78a8f0ef452498a4b45c1e29b0c7 [id=de74a6a1a22d4f61a24cd3c1c736a428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.920540 896 tablet_service.cc:1505] Processing CreateTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 (DEFAULT_TABLE table=default.Table_9e95376692ff46ceb83308b096d8cd24 [id=9dbe133fa4784c68b4410c25ed226a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.920826 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d88c516e14e62b7f491656c59b75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.920881 890 tablet_service.cc:1505] Processing CreateTablet for tablet f442dd52c7ea458ab5d8d02c70443741 (DEFAULT_TABLE table=default.Table_dd3a01ef1bab4ac1b129065e647ffb2c [id=9ee23f9b1251444089b8bb26305fea8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.921264 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442dd52c7ea458ab5d8d02c70443741. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.924229 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18e0e01fdc54b8dab97ec339d2aaa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.925233 761 tablet_service.cc:1505] Processing CreateTablet for tablet c8d40479b421435eada7682fd7f745b2 (DEFAULT_TABLE table=default.Table_b37a9d683d684befa22f0c33ef953739 [id=b0d8f7f22e8d44039d8ee280b893dc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.925282 747 tablet_service.cc:1505] Processing CreateTablet for tablet f01d88c516e14e62b7f491656c59b75a (DEFAULT_TABLE table=default.Table_bdbb78a8f0ef452498a4b45c1e29b0c7 [id=de74a6a1a22d4f61a24cd3c1c736a428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.925680 751 tablet_service.cc:1505] Processing CreateTablet for tablet f7b89ced286b4a04a28022917b1d0235 (DEFAULT_TABLE table=default.Table_a3fb78e6a50342688581081eb505f8ee [id=215977502aab462a803071559b8021a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.926483 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d40479b421435eada7682fd7f745b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.928438 763 tablet_service.cc:1505] Processing CreateTablet for tablet 8a76f229c6264d4ea65c5b9733465a62 (DEFAULT_TABLE table=default.Table_051b538b21fe4b6b8ea44de581730151 [id=56f690e90bec427f970ff0d8ef7faf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.928668 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7e050a99735b4a27a96633bad5587e4b" } new_table_name: "default.Table_f0495c28735e41e2b08d71fc162f7477" modify_external_catalogs: true
I20251029 08:52:12.928906 764 tablet_service.cc:1505] Processing CreateTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 (DEFAULT_TABLE table=default.Table_9e95376692ff46ceb83308b096d8cd24 [id=9dbe133fa4784c68b4410c25ed226a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.930759 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_183dac6abe3b4b6d8ef9614f941af3a0" } modify_external_catalogs: true
I20251029 08:52:12.930958 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_183dac6abe3b4b6d8ef9614f941af3a0" } modify_external_catalogs: true
I20251029 08:52:12.930315 1178 raft_consensus.cc:359] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.932569 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 64f73597d1764540bcaf472d3e890ee8 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:12.932648 1178 raft_consensus.cc:385] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:12.933677 1178 raft_consensus.cc:740] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:12.934597 762 tablet_service.cc:1505] Processing CreateTablet for tablet f442dd52c7ea458ab5d8d02c70443741 (DEFAULT_TABLE table=default.Table_dd3a01ef1bab4ac1b129065e647ffb2c [id=9ee23f9b1251444089b8bb26305fea8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.937422 902 tablet_service.cc:1505] Processing CreateTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa (DEFAULT_TABLE table=default.Table_13df179414624614b1663578c3d515be [id=7dbd3d93cf7a4c1f9d0abbfcba7cdf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.939090 756 tablet_service.cc:1505] Processing CreateTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa (DEFAULT_TABLE table=default.Table_13df179414624614b1663578c3d515be [id=7dbd3d93cf7a4c1f9d0abbfcba7cdf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.939724 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98324d4ae1af45fa94ae438dc2a64aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.940488 1047 tablet_service.cc:1505] Processing CreateTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 (DEFAULT_TABLE table=default.Table_9e95376692ff46ceb83308b096d8cd24 [id=9dbe133fa4784c68b4410c25ed226a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.940907 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18e0e01fdc54b8dab97ec339d2aaa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.942730 1039 tablet_service.cc:1505] Processing CreateTablet for tablet f442dd52c7ea458ab5d8d02c70443741 (DEFAULT_TABLE table=default.Table_dd3a01ef1bab4ac1b129065e647ffb2c [id=9ee23f9b1251444089b8bb26305fea8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.943070 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442dd52c7ea458ab5d8d02c70443741. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.943761 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d40479b421435eada7682fd7f745b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.944156 1044 tablet_service.cc:1505] Processing CreateTablet for tablet f01d88c516e14e62b7f491656c59b75a (DEFAULT_TABLE table=default.Table_bdbb78a8f0ef452498a4b45c1e29b0c7 [id=de74a6a1a22d4f61a24cd3c1c736a428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.944355 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b89ced286b4a04a28022917b1d0235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.944518 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d88c516e14e62b7f491656c59b75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.944808 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d88c516e14e62b7f491656c59b75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.945068 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 8a76f229c6264d4ea65c5b9733465a62 (DEFAULT_TABLE table=default.Table_051b538b21fe4b6b8ea44de581730151 [id=56f690e90bec427f970ff0d8ef7faf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.945191 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a76f229c6264d4ea65c5b9733465a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.945416 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a76f229c6264d4ea65c5b9733465a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.945583 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18e0e01fdc54b8dab97ec339d2aaa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.945981 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98324d4ae1af45fa94ae438dc2a64aaa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.946417 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_e4e823c7272f44e1af831ef60277d505 [id=93bd45834be24532b19ed170c43ac0dc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8a9f0b579ebd4d2db813459611d7f9bf consensus state has no leader
I20251029 08:52:12.946689 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442dd52c7ea458ab5d8d02c70443741. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.946806 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 468) took 1765 ms. Trace:
I20251029 08:52:12.946952 1179 rpcz_store.cc:276] 1029 08:52:11.180767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.242153 (+ 61386us) service_pool.cc:225] Handling call
1029 08:52:12.736424 (+1494271us) ts_tablet_manager.cc:1151] Deleting tablet a7fdbe6681894ffa8264cdfa486f16e6
1029 08:52:12.736434 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.802153 (+ 65719us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.874541 (+ 72388us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:12.946700 (+ 72159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494030}
I20251029 08:52:12.947728 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 475) took 1644 ms. Trace:
I20251029 08:52:12.947870 1179 rpcz_store.cc:276] 1029 08:52:11.303668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.303858 (+ 190us) service_pool.cc:225] Handling call
1029 08:52:12.947428 (+1643570us) ts_tablet_manager.cc:1151] Deleting tablet be706771a95e45e5a20a15614eadfe1a
1029 08:52:12.947460 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.947630 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643332}
I20251029 08:52:12.948537 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 476) took 1643 ms. Trace:
I20251029 08:52:12.948674 1179 rpcz_store.cc:276] 1029 08:52:11.304943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.305144 (+ 201us) service_pool.cc:225] Handling call
1029 08:52:12.948311 (+1643167us) ts_tablet_manager.cc:1151] Deleting tablet 840ff254244f43a18679232279e02d1f
1029 08:52:12.948321 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.948462 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642847}
I20251029 08:52:12.949244 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 477) took 1536 ms. Trace:
I20251029 08:52:12.951432 896 tablet_service.cc:1505] Processing CreateTablet for tablet e867dd08ca83407abff01e927cc09e5e (DEFAULT_TABLE table=default.Table_0a20ef9c732a4c6bb45224c8a4454cf4 [id=8f3fa6ad0f0a48618ac6799de827c5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.951474 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a7fdbe6681894ffa8264cdfa486f16e6 (table default.Table_80b0f80eda77452b959592ce98b44953 [id=f65a6a353027447ba8004192ad75300f]) successfully deleted
W20251029 08:52:12.951753 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet be706771a95e45e5a20a15614eadfe1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be706771a95e45e5a20a15614eadfe1a already in progress: creating tablet
I20251029 08:52:12.951851 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e867dd08ca83407abff01e927cc09e5e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:12.952008 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 840ff254244f43a18679232279e02d1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840ff254244f43a18679232279e02d1f already in progress: creating tablet
I20251029 08:52:12.953167 1045 tablet_service.cc:1505] Processing CreateTablet for tablet c8d40479b421435eada7682fd7f745b2 (DEFAULT_TABLE table=default.Table_b37a9d683d684befa22f0c33ef953739 [id=b0d8f7f22e8d44039d8ee280b893dc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.955986 1040 tablet_service.cc:1505] Processing CreateTablet for tablet f7b89ced286b4a04a28022917b1d0235 (DEFAULT_TABLE table=default.Table_a3fb78e6a50342688581081eb505f8ee [id=215977502aab462a803071559b8021a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.956303 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa (DEFAULT_TABLE table=default.Table_13df179414624614b1663578c3d515be [id=7dbd3d93cf7a4c1f9d0abbfcba7cdf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.957406 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d40479b421435eada7682fd7f745b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.962278 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 387) took 3425 ms. Trace:
I20251029 08:52:12.962415 1175 rpcz_store.cc:276] 1029 08:52:09.536703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.536755 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:12.962091 (+3425336us) ts_tablet_manager.cc:1151] Deleting tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
1029 08:52:12.962105 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.962234 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3414640}
I20251029 08:52:12.962778 1032 tablet_service.cc:1505] Processing CreateTablet for tablet e867dd08ca83407abff01e927cc09e5e (DEFAULT_TABLE table=default.Table_0a20ef9c732a4c6bb45224c8a4454cf4 [id=8f3fa6ad0f0a48618ac6799de827c5d0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:12.962990 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa0ff14e19b4ad7a8b33796481f0f3f already in progress: creating tablet
I20251029 08:52:12.963702 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98324d4ae1af45fa94ae438dc2a64aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.963975 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_745a04b358a243b4a7f0e0c7f3526da7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.964115 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 388) took 3423 ms. Trace:
I20251029 08:52:12.964227 1175 rpcz_store.cc:276] 1029 08:52:09.540530 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.540578 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:12.963952 (+3423374us) ts_tablet_manager.cc:1151] Deleting tablet 5385fef404a4463595074f0c8ff822d7
1029 08:52:12.963964 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.964077 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3405239}
W20251029 08:52:12.964596 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_745a04b358a243b4a7f0e0c7f3526da7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:12.964694 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5385fef404a4463595074f0c8ff822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5385fef404a4463595074f0c8ff822d7 already in progress: creating tablet
I20251029 08:52:12.965126 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e867dd08ca83407abff01e927cc09e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.965332 760 tablet_service.cc:1505] Processing CreateTablet for tablet e867dd08ca83407abff01e927cc09e5e (DEFAULT_TABLE table=default.Table_0a20ef9c732a4c6bb45224c8a4454cf4 [id=8f3fa6ad0f0a48618ac6799de827c5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.965540 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b89ced286b4a04a28022917b1d0235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.965703 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e867dd08ca83407abff01e927cc09e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.966399 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 389) took 3409 ms. Trace:
I20251029 08:52:12.966516 1175 rpcz_store.cc:276] 1029 08:52:09.556779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.560483 (+ 3704us) service_pool.cc:225] Handling call
1029 08:52:12.966248 (+3405765us) ts_tablet_manager.cc:1151] Deleting tablet 2fb380018f46465e82ac6223ecf8dd0d
1029 08:52:12.966259 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.966364 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390348}
W20251029 08:52:12.967010 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2fb380018f46465e82ac6223ecf8dd0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb380018f46465e82ac6223ecf8dd0d already in progress: creating tablet
I20251029 08:52:12.970598 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 390) took 3409 ms. Trace:
I20251029 08:52:12.970713 1175 rpcz_store.cc:276] 1029 08:52:09.560800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.561074 (+ 274us) service_pool.cc:225] Handling call
1029 08:52:12.970431 (+3409357us) ts_tablet_manager.cc:1151] Deleting tablet bc96ef1a94844a32923087c44a5bcfc8
1029 08:52:12.970442 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.970551 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3394403}
W20251029 08:52:12.971174 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bc96ef1a94844a32923087c44a5bcfc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc96ef1a94844a32923087c44a5bcfc8 already in progress: creating tablet
I20251029 08:52:12.972329 892 tablet_service.cc:1505] Processing CreateTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.972479 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1f5f2cd9e3f14a0a8f3316489f46929a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:12.972882 759 tablet_service.cc:1505] Processing CreateTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:12.973057 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f5f2cd9e3f14a0a8f3316489f46929a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:12.973336 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13afd85e234643e08957e7a7f7fd7e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.973783 755 tablet_service.cc:1505] Processing CreateTablet for tablet 739f3ef68dc141af90954bf0da4c5d2e (DEFAULT_TABLE table=default.Table_8ec38d64e3d3455ab090035a692b37ff [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.974135 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739f3ef68dc141af90954bf0da4c5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.980435 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 397) took 3134 ms. Trace:
I20251029 08:52:12.980590 1175 rpcz_store.cc:276] 1029 08:52:09.845682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:09.847824 (+ 2142us) service_pool.cc:225] Handling call
1029 08:52:12.980249 (+3132425us) ts_tablet_manager.cc:1151] Deleting tablet 8991ea78321545be85d5c480841e4751
1029 08:52:12.980262 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.980390 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099847}
W20251029 08:52:12.981122 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8991ea78321545be85d5c480841e4751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8991ea78321545be85d5c480841e4751 already in progress: creating tablet
I20251029 08:52:12.982348 1175 tablet_replica.cc:333] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:12.982586 1175 raft_consensus.cc:2243] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:12.982789 1175 raft_consensus.cc:2272] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:12.983830 1774 raft_consensus.cc:493] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:12.983984 1774 raft_consensus.cc:515] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:12.984277 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:12.984688 1175 ts_tablet_manager.cc:1916] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:12.984716 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13afd85e234643e08957e7a7f7fd7e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:12.984926 1774 leader_election.cc:290] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.985868 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:12.986096 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:12.986495 718 leader_election.cc:343] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:12.988243 720 leader_election.cc:343] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:12.988416 720 leader_election.cc:304] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:12.988933 1774 raft_consensus.cc:2749] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.001340 893 tablet_service.cc:1505] Processing CreateTablet for tablet 739f3ef68dc141af90954bf0da4c5d2e (DEFAULT_TABLE table=default.Table_8ec38d64e3d3455ab090035a692b37ff [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.001987 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13afd85e234643e08957e7a7f7fd7e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.002221 890 tablet_service.cc:1505] Processing CreateTablet for tablet a186df0db24c4d22abee7ec57bcb199b (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.002641 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739f3ef68dc141af90954bf0da4c5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.009951 888 tablet_service.cc:1505] Processing CreateTablet for tablet e3a62daad378430c9b593276685fa228 (DEFAULT_TABLE table=default.Table_ab04114f072046e2a09d1d5559571e7b [id=61ec0ef421304c1090332d3c8f120370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.010537 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a62daad378430c9b593276685fa228. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.010553 897 tablet_service.cc:1505] Processing CreateTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 (DEFAULT_TABLE table=default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e [id=77c425795492472d9a3943a8301a745f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.011119 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64058f9cb3ad4844ac87bc0cc27774b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.012562 1050 tablet_service.cc:1505] Processing CreateTablet for tablet a186df0db24c4d22abee7ec57bcb199b (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.013218 1037 tablet_service.cc:1505] Processing CreateTablet for tablet e3a62daad378430c9b593276685fa228 (DEFAULT_TABLE table=default.Table_ab04114f072046e2a09d1d5559571e7b [id=61ec0ef421304c1090332d3c8f120370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.013811 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a186df0db24c4d22abee7ec57bcb199b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.016821 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b (DEFAULT_TABLE table=default.Table_87bd0ffa8ae04a5da1b10cae0b440517 [id=00b9b444202e4acd94df75944ae69619]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.017189 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.019491 894 tablet_service.cc:1505] Processing CreateTablet for tablet 52b697b962f849eb88741c2e2ba4643d (DEFAULT_TABLE table=default.Table_3e9067e3526a433e8b4d64a6a5b739a5 [id=cc4d21d0018049b4bd3d3387e4fcdfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.019855 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b697b962f849eb88741c2e2ba4643d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.022285 763 tablet_service.cc:1505] Processing CreateTablet for tablet e3a62daad378430c9b593276685fa228 (DEFAULT_TABLE table=default.Table_ab04114f072046e2a09d1d5559571e7b [id=61ec0ef421304c1090332d3c8f120370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.022866 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a62daad378430c9b593276685fa228. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.026130 898 tablet_service.cc:1505] Processing CreateTablet for tablet 8e1084a2b4c44c72a5904cf71b9e19af (DEFAULT_TABLE table=default.Table_69a998c83a8c499d86a9931423a1c53f [id=4ca9257c30794549b56af097c1818f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.026222 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a62daad378430c9b593276685fa228. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.026521 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e1084a2b4c44c72a5904cf71b9e19af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.027258 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 (DEFAULT_TABLE table=default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e [id=77c425795492472d9a3943a8301a745f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.027608 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 739f3ef68dc141af90954bf0da4c5d2e (DEFAULT_TABLE table=default.Table_8ec38d64e3d3455ab090035a692b37ff [id=58aabf0c50d64caa918b4d97574d7c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.027688 759 tablet_service.cc:1505] Processing CreateTablet for tablet aba71882ee5d41f5a5a007eba227c279 (DEFAULT_TABLE table=default.Table_c68edbe3428b41579ae051abdb61b40d [id=6df59fd39c8b452493105773284b0e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.027977 764 tablet_service.cc:1505] Processing CreateTablet for tablet 52b697b962f849eb88741c2e2ba4643d (DEFAULT_TABLE table=default.Table_3e9067e3526a433e8b4d64a6a5b739a5 [id=cc4d21d0018049b4bd3d3387e4fcdfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.028208 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba71882ee5d41f5a5a007eba227c279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.028529 747 tablet_service.cc:1505] Processing CreateTablet for tablet a186df0db24c4d22abee7ec57bcb199b (DEFAULT_TABLE table=default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.028887 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a186df0db24c4d22abee7ec57bcb199b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.030406 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b (DEFAULT_TABLE table=default.Table_87bd0ffa8ae04a5da1b10cae0b440517 [id=00b9b444202e4acd94df75944ae69619]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.032327 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.032832 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 52b697b962f849eb88741c2e2ba4643d (DEFAULT_TABLE table=default.Table_3e9067e3526a433e8b4d64a6a5b739a5 [id=cc4d21d0018049b4bd3d3387e4fcdfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.034090 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8e1084a2b4c44c72a5904cf71b9e19af (DEFAULT_TABLE table=default.Table_69a998c83a8c499d86a9931423a1c53f [id=4ca9257c30794549b56af097c1818f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.027685 761 tablet_service.cc:1505] Processing CreateTablet for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b (DEFAULT_TABLE table=default.Table_87bd0ffa8ae04a5da1b10cae0b440517 [id=00b9b444202e4acd94df75944ae69619]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.034453 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b697b962f849eb88741c2e2ba4643d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.038082 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721f0ab79d64b24a7b8b4797a5a3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.038129 1774 raft_consensus.cc:493] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.038302 1774 raft_consensus.cc:515] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.038972 1774 leader_election.cc:290] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.042721 759 tablet_service.cc:1505] Processing CreateTablet for tablet 90360621ce124475af2f12060471f63c (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.043162 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90360621ce124475af2f12060471f63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.045483 762 tablet_service.cc:1505] Processing CreateTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 (DEFAULT_TABLE table=default.Table_23cbf2fd266c45efbdb3447e52d13f9a [id=44088afcca98466684bebaf1cc7094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.046482 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49b69d8e23446a88c9e28445a8b4b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.048172 886 tablet_service.cc:1505] Processing CreateTablet for tablet 90360621ce124475af2f12060471f63c (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.048216 756 tablet_service.cc:1505] Processing CreateTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 (DEFAULT_TABLE table=default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e [id=77c425795492472d9a3943a8301a745f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.048619 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90360621ce124475af2f12060471f63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.049469 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b697b962f849eb88741c2e2ba4643d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.050398 899 tablet_service.cc:1505] Processing CreateTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 (DEFAULT_TABLE table=default.Table_23cbf2fd266c45efbdb3447e52d13f9a [id=44088afcca98466684bebaf1cc7094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.050791 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49b69d8e23446a88c9e28445a8b4b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.051183 891 tablet_service.cc:1505] Processing CreateTablet for tablet 82c522c0b20647b296d0439773a95aa6 (DEFAULT_TABLE table=default.Table_54c3fb69aa194818946513e7b678f20a [id=e7e202692e8444029df359c9a772e0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.051713 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c522c0b20647b296d0439773a95aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.051764 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739f3ef68dc141af90954bf0da4c5d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.052438 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 8e1084a2b4c44c72a5904cf71b9e19af (DEFAULT_TABLE table=default.Table_69a998c83a8c499d86a9931423a1c53f [id=4ca9257c30794549b56af097c1818f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.027581 895 tablet_service.cc:1505] Processing CreateTablet for tablet aba71882ee5d41f5a5a007eba227c279 (DEFAULT_TABLE table=default.Table_c68edbe3428b41579ae051abdb61b40d [id=6df59fd39c8b452493105773284b0e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.054670 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba71882ee5d41f5a5a007eba227c279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.054886 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e1084a2b4c44c72a5904cf71b9e19af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.056481 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.057427 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64058f9cb3ad4844ac87bc0cc27774b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:13.068336 718 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.069268 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a186df0db24c4d22abee7ec57bcb199b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.063868 761 tablet_service.cc:1505] Processing CreateTablet for tablet 82c522c0b20647b296d0439773a95aa6 (DEFAULT_TABLE table=default.Table_54c3fb69aa194818946513e7b678f20a [id=e7e202692e8444029df359c9a772e0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.075083 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c522c0b20647b296d0439773a95aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.076242 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e1084a2b4c44c72a5904cf71b9e19af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.080178 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64058f9cb3ad4844ac87bc0cc27774b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.089403 1483 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" instance_seqno: 1761727920815568) as {username='slave'} at 127.0.104.2:45495; Asking this server to re-register.
I20251029 08:52:13.090731 972 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:13.091040 1481 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" instance_seqno: 1761727920315454) as {username='slave'} at 127.0.104.1:46445; Asking this server to re-register.
I20251029 08:52:13.091679 1479 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" instance_seqno: 1761727921574165) as {username='slave'} at 127.0.104.3:44889; Asking this server to re-register.
I20251029 08:52:13.092578 1117 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:13.093184 830 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:13.095614 1479 ts_manager.cc:194] Registered new tserver with Master: 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.097749 1481 ts_manager.cc:194] Registered new tserver with Master: 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:13.099110 1481 ts_manager.cc:194] Registered new tserver with Master: 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:13.109133 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 82c522c0b20647b296d0439773a95aa6 (DEFAULT_TABLE table=default.Table_54c3fb69aa194818946513e7b678f20a [id=e7e202692e8444029df359c9a772e0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.119362 1038 tablet_service.cc:1505] Processing CreateTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 (DEFAULT_TABLE table=default.Table_23cbf2fd266c45efbdb3447e52d13f9a [id=44088afcca98466684bebaf1cc7094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.133598 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 90360621ce124475af2f12060471f63c (DEFAULT_TABLE table=default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.134218 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:13.134630 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c522c0b20647b296d0439773a95aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.134907 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49b69d8e23446a88c9e28445a8b4b56. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:13.143482 720 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.143685 720 leader_election.cc:304] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.144357 1774 raft_consensus.cc:2749] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.144991 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90360621ce124475af2f12060471f63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.147903 1791 raft_consensus.cc:493] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:13.148126 1791 raft_consensus.cc:515] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:13.149266 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 712 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:52:13.151259 1175 ts_tablet_manager.cc:1929] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:13.151461 1047 tablet_service.cc:1505] Processing CreateTablet for tablet aba71882ee5d41f5a5a007eba227c279 (DEFAULT_TABLE table=default.Table_c68edbe3428b41579ae051abdb61b40d [id=6df59fd39c8b452493105773284b0e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.151995 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba71882ee5d41f5a5a007eba227c279. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:13.153298 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 62635.6us
I20251029 08:52:13.156123 1175 log.cc:1199] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/57e2b19a227740a28ae601d77e62597b
I20251029 08:52:13.156711 1175 ts_tablet_manager.cc:1950] T 57e2b19a227740a28ae601d77e62597b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.161244 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 712 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:13.161495 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 420) took 2990 ms. Trace:
I20251029 08:52:13.161893 1451 leader_election.cc:304] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, 85c8cdf28bda444d851fb50588400173
I20251029 08:52:13.163393 591 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734] alter complete (version 2)
I20251029 08:52:13.165961 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_70d88123dfe248b6aeb0e6fa51fafc49" } modify_external_catalogs: true
I20251029 08:52:13.166200 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_70d88123dfe248b6aeb0e6fa51fafc49" } modify_external_catalogs: true
I20251029 08:52:13.173892 1791 leader_election.cc:290] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:13.174075 1791 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251029 08:52:13.161631 1175 rpcz_store.cc:276] 1029 08:52:10.170708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.238231 (+ 67523us) service_pool.cc:225] Handling call
1029 08:52:12.982276 (+2744045us) ts_tablet_manager.cc:1151] Deleting tablet 57e2b19a227740a28ae601d77e62597b
1029 08:52:12.982287 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.136465 (+154178us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.151121 (+ 14656us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.161430 (+ 10309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2743843}
I20251029 08:52:13.175848 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 57e2b19a227740a28ae601d77e62597b (table default.Table_ab2c1cce4615467595cb78eb6ca1b4ec [id=d283eaf060754468a34dfd1849f618ad]) successfully deleted
I20251029 08:52:13.178462 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 421) took 3007 ms. Trace:
I20251029 08:52:13.178659 1175 rpcz_store.cc:276] 1029 08:52:10.170863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.238449 (+ 67586us) service_pool.cc:225] Handling call
1029 08:52:13.176786 (+2938337us) ts_tablet_manager.cc:1151] Deleting tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
1029 08:52:13.176798 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.178392 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2938165}
W20251029 08:52:13.179208 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 already in progress: creating tablet
I20251029 08:52:13.179718 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 422) took 3008 ms. Trace:
I20251029 08:52:13.183449 1175 rpcz_store.cc:276] 1029 08:52:10.173613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.238633 (+ 65020us) service_pool.cc:225] Handling call
1029 08:52:13.179531 (+2940898us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:13.179542 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.179665 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940727}
W20251029 08:52:13.184433 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet eed66c3fa3f848df905a2a31fd553474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eed66c3fa3f848df905a2a31fd553474 already in progress: creating tablet
I20251029 08:52:13.185024 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 423) took 3011 ms. Trace:
I20251029 08:52:13.185833 1175 rpcz_store.cc:276] 1029 08:52:10.173758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.238812 (+ 65054us) service_pool.cc:225] Handling call
1029 08:52:13.184854 (+2946042us) ts_tablet_manager.cc:1151] Deleting tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
1029 08:52:13.184864 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.184958 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945892}
W20251029 08:52:13.186584 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d already in progress: creating tablet
I20251029 08:52:13.187048 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 424) took 3013 ms. Trace:
I20251029 08:52:13.187192 1175 rpcz_store.cc:276] 1029 08:52:10.173848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.238972 (+ 65124us) service_pool.cc:225] Handling call
1029 08:52:13.186887 (+2947915us) ts_tablet_manager.cc:1151] Deleting tablet bf29e030892d44e1ad4d1ca4f5aeab3b
1029 08:52:13.186896 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.186989 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2947779}
W20251029 08:52:13.187688 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bf29e030892d44e1ad4d1ca4f5aeab3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf29e030892d44e1ad4d1ca4f5aeab3b already in progress: creating tablet
I20251029 08:52:13.188211 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 425) took 3014 ms. Trace:
I20251029 08:52:13.188351 1175 rpcz_store.cc:276] 1029 08:52:10.173957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.239121 (+ 65164us) service_pool.cc:225] Handling call
1029 08:52:13.187963 (+2948842us) ts_tablet_manager.cc:1151] Deleting tablet 14b665b63a7341d699d499ffc8cfd9f7
1029 08:52:13.187972 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.188151 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2948693}
W20251029 08:52:13.189123 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 14b665b63a7341d699d499ffc8cfd9f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b665b63a7341d699d499ffc8cfd9f7 already in progress: creating tablet
I20251029 08:52:13.189553 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 439) took 2566 ms. Trace:
I20251029 08:52:13.189706 1175 rpcz_store.cc:276] 1029 08:52:10.623061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.623174 (+ 113us) service_pool.cc:225] Handling call
1029 08:52:13.189401 (+2566227us) ts_tablet_manager.cc:1151] Deleting tablet fa8884bb8e64474d936c8825d1ef3545
1029 08:52:13.189410 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.189494 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2562281}
W20251029 08:52:13.190199 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fa8884bb8e64474d936c8825d1ef3545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa8884bb8e64474d936c8825d1ef3545 already in progress: creating tablet
I20251029 08:52:13.190636 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 441) took 2567 ms. Trace:
I20251029 08:52:13.190779 1175 rpcz_store.cc:276] 1029 08:52:10.623292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.623338 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:13.190475 (+2567137us) ts_tablet_manager.cc:1151] Deleting tablet add0605edf5c4511ae4c8d9cb7f0babb
1029 08:52:13.190485 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.190579 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2558411}
W20251029 08:52:13.191241 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet add0605edf5c4511ae4c8d9cb7f0babb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet add0605edf5c4511ae4c8d9cb7f0babb already in progress: creating tablet
I20251029 08:52:13.191679 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 440) took 2568 ms. Trace:
I20251029 08:52:13.191813 1175 rpcz_store.cc:276] 1029 08:52:10.623180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.624193 (+ 1013us) service_pool.cc:225] Handling call
1029 08:52:13.191508 (+2567315us) ts_tablet_manager.cc:1151] Deleting tablet ba1bcb3f6c134de2b322696501927a16
1029 08:52:13.191517 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.191621 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2551321}
W20251029 08:52:13.192307 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ba1bcb3f6c134de2b322696501927a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba1bcb3f6c134de2b322696501927a16 already in progress: creating tablet
I20251029 08:52:13.192752 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 437) took 2569 ms. Trace:
I20251029 08:52:13.193182 1175 rpcz_store.cc:276] 1029 08:52:10.622840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.624370 (+ 1530us) service_pool.cc:225] Handling call
1029 08:52:13.192583 (+2568213us) ts_tablet_manager.cc:1151] Deleting tablet 2f40aafe7dc94851a67abd84d80d7d7b
1029 08:52:13.192592 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.192694 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2545503}
W20251029 08:52:13.193890 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2f40aafe7dc94851a67abd84d80d7d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f40aafe7dc94851a67abd84d80d7d7b already in progress: creating tablet
I20251029 08:52:13.195084 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 438) took 2572 ms. Trace:
I20251029 08:52:13.195497 1175 rpcz_store.cc:276] 1029 08:52:10.622938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.624538 (+ 1600us) service_pool.cc:225] Handling call
1029 08:52:13.194939 (+2570401us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:13.194947 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.195025 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2546844}
W20251029 08:52:13.196182 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5ac1af8b78a4d94b0e3da14ebfffd92 already in progress: creating tablet
I20251029 08:52:13.196846 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 449) took 2572 ms. Trace:
I20251029 08:52:13.196988 1175 rpcz_store.cc:276] 1029 08:52:10.624198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.648120 (+ 23922us) service_pool.cc:225] Handling call
1029 08:52:13.196700 (+2548580us) ts_tablet_manager.cc:1151] Deleting tablet 7a355c68ecfb45ff8c396873d91268a8
1029 08:52:13.196707 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.196785 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548374}
W20251029 08:52:13.197521 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7a355c68ecfb45ff8c396873d91268a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a355c68ecfb45ff8c396873d91268a8 already in progress: creating tablet
I20251029 08:52:13.197953 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 450) took 2573 ms. Trace:
I20251029 08:52:13.198091 1175 rpcz_store.cc:276] 1029 08:52:10.624326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.648348 (+ 24022us) service_pool.cc:225] Handling call
1029 08:52:13.197792 (+2549444us) ts_tablet_manager.cc:1151] Deleting tablet 7b5051ff776e4975b917db0393a1f945
1029 08:52:13.197800 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.197898 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2549300}
W20251029 08:52:13.198582 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7b5051ff776e4975b917db0393a1f945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b5051ff776e4975b917db0393a1f945 already in progress: creating tablet
I20251029 08:52:13.199463 1175 tablet_replica.cc:333] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.199731 1175 raft_consensus.cc:2243] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:13.200201 1175 raft_consensus.cc:2272] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.200887 1175 ts_tablet_manager.cc:1916] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:13.201853 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c18e0e01fdc54b8dab97ec339d2aaa07 consensus state has no leader
I20251029 08:52:13.202222 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_08e6f2532b724e02acbd01a2d18fb7a0" } modify_external_catalogs: true
I20251029 08:52:13.202425 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_08e6f2532b724e02acbd01a2d18fb7a0" } modify_external_catalogs: true
I20251029 08:52:13.204051 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7046bb1fcdd34d308aecddfdf0752911
I20251029 08:52:13.204857 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_32a7c0c8b359467995851e70b6448f62" } new_table_name: "default.Table_3fffb85f46a84893b72083c6894e6f84" modify_external_catalogs: true
I20251029 08:52:13.205590 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.206377 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_01fe9fda4b2c432cba9c90b4f48f52aa" } new_table_name: "default.Table_afe756d752d549e6835f0f24b39e3398" modify_external_catalogs: true
I20251029 08:52:13.206848 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.209439 1177 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.209614 1177 ts_tablet_manager.cc:1403] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.314s user 0.005s sys 0.023s
I20251029 08:52:12.949378 1179 rpcz_store.cc:276] 1029 08:52:11.413238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.413300 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:12.949005 (+1535705us) ts_tablet_manager.cc:1151] Deleting tablet e0fa089fce564af4b03aa323cec6cbf6
1029 08:52:12.949038 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:12.949171 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535469}
I20251029 08:52:13.210505 1177 raft_consensus.cc:359] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.210696 1177 raft_consensus.cc:385] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.210805 1177 raft_consensus.cc:740] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.211063 1177 consensus_queue.cc:260] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.211606 1177 ts_tablet_manager.cc:1434] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:13.211908 1177 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.217731 1177 tablet_bootstrap.cc:654] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.220940 1175 ts_tablet_manager.cc:1929] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20251029 08:52:13.222333 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e0fa089fce564af4b03aa323cec6cbf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fa089fce564af4b03aa323cec6cbf6 already in progress: creating tablet
I20251029 08:52:13.222924 1175 log.cc:1199] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aea79bd4e6844b7abac623824cabbeeb
I20251029 08:52:13.223398 1175 ts_tablet_manager.cc:1950] T aea79bd4e6844b7abac623824cabbeeb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.225520 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 426) took 2804 ms. Trace:
I20251029 08:52:13.225698 1175 rpcz_store.cc:276] 1029 08:52:10.437376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.692421 (+255045us) service_pool.cc:225] Handling call
1029 08:52:13.198844 (+2506423us) ts_tablet_manager.cc:1151] Deleting tablet aea79bd4e6844b7abac623824cabbeeb
1029 08:52:13.198853 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.207219 (+ 8366us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.220785 (+ 13566us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.225456 (+ 4671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2471966}
I20251029 08:52:13.227384 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 464) took 2563 ms. Trace:
I20251029 08:52:13.227533 1751 raft_consensus.cc:493] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.227516 1175 rpcz_store.cc:276] 1029 08:52:10.664192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.777910 (+113718us) service_pool.cc:225] Handling call
1029 08:52:13.227216 (+2449306us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:13.227226 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.227343 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2437725}
W20251029 08:52:13.228040 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 657cdaad97e24611a997e26a6293ea5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657cdaad97e24611a997e26a6293ea5e already in progress: creating tablet
I20251029 08:52:13.229141 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 465) took 2564 ms. Trace:
I20251029 08:52:13.229626 1175 rpcz_store.cc:276] 1029 08:52:10.664277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.783322 (+119045us) service_pool.cc:225] Handling call
1029 08:52:13.228988 (+2445666us) ts_tablet_manager.cc:1151] Deleting tablet b7da7e08b74f4a9aacedada42e839a11
1029 08:52:13.228998 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.229100 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2306289}
W20251029 08:52:13.230399 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b7da7e08b74f4a9aacedada42e839a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7da7e08b74f4a9aacedada42e839a11 already in progress: creating tablet
I20251029 08:52:13.230931 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 466) took 2564 ms. Trace:
I20251029 08:52:13.231388 1175 rpcz_store.cc:276] 1029 08:52:10.666123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.789529 (+123406us) service_pool.cc:225] Handling call
1029 08:52:13.230785 (+2441256us) ts_tablet_manager.cc:1151] Deleting tablet aa411391c63146e7acb8b4c6edbe12a0
1029 08:52:13.230795 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.230890 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2286307}
W20251029 08:52:13.232141 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa411391c63146e7acb8b4c6edbe12a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa411391c63146e7acb8b4c6edbe12a0 already in progress: creating tablet
I20251029 08:52:13.232523 1751 raft_consensus.cc:515] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.232647 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 467) took 2566 ms. Trace:
I20251029 08:52:13.232755 1175 rpcz_store.cc:276] 1029 08:52:10.666575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.922747 (+256172us) service_pool.cc:225] Handling call
1029 08:52:13.232466 (+2309719us) ts_tablet_manager.cc:1151] Deleting tablet aa4f97075b3941aaa7cda3f9413e867f
1029 08:52:13.232476 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.232610 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2286828}
W20251029 08:52:13.233158 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa4f97075b3941aaa7cda3f9413e867f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa4f97075b3941aaa7cda3f9413e867f already in progress: creating tablet
I20251029 08:52:13.233773 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.234457 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 468) took 2567 ms. Trace:
I20251029 08:52:13.234593 1175 rpcz_store.cc:276] 1029 08:52:10.666694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:10.944522 (+277828us) service_pool.cc:225] Handling call
1029 08:52:13.234278 (+2289756us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:13.234285 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.234417 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2284985}
W20251029 08:52:13.234982 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aafa846cb2144cec9ce4b28e61d91961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafa846cb2144cec9ce4b28e61d91961 already in progress: creating tablet
I20251029 08:52:13.235890 1177 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.236136 1177 ts_tablet_manager.cc:1403] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20251029 08:52:13.237658 1177 raft_consensus.cc:359] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.237944 1177 raft_consensus.cc:385] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.238080 1177 raft_consensus.cc:740] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.238420 1177 consensus_queue.cc:260] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.238639 1183 tablet_bootstrap.cc:492] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:13.238830 1183 ts_tablet_manager.cc:1403] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.440s user 0.007s sys 0.003s
I20251029 08:52:13.239634 1183 raft_consensus.cc:359] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.239858 1183 raft_consensus.cc:385] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.239974 1183 raft_consensus.cc:740] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.240336 1183 consensus_queue.cc:260] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.242190 1177 ts_tablet_manager.cc:1434] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:13.242622 1177 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.246554 1183 ts_tablet_manager.cc:1434] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 08:52:13.242946 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 475) took 2561 ms. Trace:
I20251029 08:52:13.248054 1175 rpcz_store.cc:276] 1029 08:52:10.681349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.015887 (+334538us) service_pool.cc:225] Handling call
1029 08:52:13.242789 (+2226902us) ts_tablet_manager.cc:1151] Deleting tablet 39fc9612689b46d498d6092bd552da4e
1029 08:52:13.242801 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.242915 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2179587}
W20251029 08:52:13.248534 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 39fc9612689b46d498d6092bd552da4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39fc9612689b46d498d6092bd552da4e already in progress: creating tablet
I20251029 08:52:13.248558 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 489) took 1814 ms. Trace:
I20251029 08:52:13.248662 1179 rpcz_store.cc:276] 1029 08:52:11.417379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.428094 (+ 10715us) service_pool.cc:225] Handling call
1029 08:52:13.222674 (+1794580us) ts_tablet_manager.cc:1151] Deleting tablet 608b7095d42941ccb0c4eb57f6754e74
1029 08:52:13.222684 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.232333 (+ 9649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794373}
I20251029 08:52:13.249315 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 477) took 2207 ms. Trace:
I20251029 08:52:13.249442 1175 rpcz_store.cc:276] 1029 08:52:11.043138 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.061262 (+ 18124us) service_pool.cc:225] Handling call
1029 08:52:13.249163 (+2187901us) ts_tablet_manager.cc:1151] Deleting tablet 43c3074caee74069a8a7f1b87721d726
1029 08:52:13.249175 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.249275 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171218}
W20251029 08:52:13.249917 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 43c3074caee74069a8a7f1b87721d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43c3074caee74069a8a7f1b87721d726 already in progress: creating tablet
I20251029 08:52:13.251390 1177 tablet_bootstrap.cc:654] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:13.251399 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 608b7095d42941ccb0c4eb57f6754e74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608b7095d42941ccb0c4eb57f6754e74 already in progress: creating tablet
I20251029 08:52:13.252559 1751 leader_election.cc:290] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.252892 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a57b1dbbdf854bc6a6e1b5dc88b63bff" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.253101 922 raft_consensus.cc:2468] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:13.251695 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 478) took 2186 ms. Trace:
I20251029 08:52:13.253489 996 leader_election.cc:304] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:13.253472 1175 rpcz_store.cc:276] 1029 08:52:11.065073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.065633 (+ 560us) service_pool.cc:225] Handling call
1029 08:52:13.251548 (+2185915us) ts_tablet_manager.cc:1151] Deleting tablet 9907d044daa145168417371c2bd66544
1029 08:52:13.251558 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.251654 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2172303}
W20251029 08:52:13.253893 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9907d044daa145168417371c2bd66544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9907d044daa145168417371c2bd66544 already in progress: creating tablet
I20251029 08:52:13.254395 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 483) took 2159 ms. Trace:
I20251029 08:52:13.254545 1175 rpcz_store.cc:276] 1029 08:52:11.095278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.096654 (+ 1376us) service_pool.cc:225] Handling call
1029 08:52:13.254272 (+2157618us) ts_tablet_manager.cc:1151] Deleting tablet 351047c725f8488abbd074bd40790b48
1029 08:52:13.254282 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.254366 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157247}
W20251029 08:52:13.255018 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 351047c725f8488abbd074bd40790b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351047c725f8488abbd074bd40790b48 already in progress: creating tablet
I20251029 08:52:13.255414 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 480) took 2184 ms. Trace:
I20251029 08:52:13.255575 1175 rpcz_store.cc:276] 1029 08:52:11.070847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.076413 (+ 5566us) service_pool.cc:225] Handling call
1029 08:52:13.255291 (+2178878us) ts_tablet_manager.cc:1151] Deleting tablet d50e2fdfc0e64516a41c90f5a56f4c89
1029 08:52:13.255300 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.255386 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153923}
W20251029 08:52:13.256062 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d50e2fdfc0e64516a41c90f5a56f4c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d50e2fdfc0e64516a41c90f5a56f4c89 already in progress: creating tablet
I20251029 08:52:13.256443 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 482) took 2185 ms. Trace:
I20251029 08:52:13.256995 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:13.257231 1175 rpcz_store.cc:276] 1029 08:52:11.071050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.077970 (+ 6920us) service_pool.cc:225] Handling call
1029 08:52:13.256327 (+2178357us) ts_tablet_manager.cc:1151] Deleting tablet e8c4d793f10147fba5e4f44474575bf0
1029 08:52:13.256336 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.256413 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154430}
I20251029 08:52:13.257843 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 481) took 2186 ms. Trace:
I20251029 08:52:13.257949 1175 rpcz_store.cc:276] 1029 08:52:11.070932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.076390 (+ 5458us) service_pool.cc:225] Handling call
1029 08:52:13.257723 (+2181333us) ts_tablet_manager.cc:1151] Deleting tablet e3c24c0d32f74435bd9d28f2415e2c47
1029 08:52:13.257732 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.257816 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2138463}
I20251029 08:52:13.258432 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 479) took 2187 ms. Trace:
I20251029 08:52:13.258528 1175 rpcz_store.cc:276] 1029 08:52:11.070683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.076346 (+ 5663us) service_pool.cc:225] Handling call
1029 08:52:13.258313 (+2181967us) ts_tablet_manager.cc:1151] Deleting tablet af38662420d14ac1a7cb5fd03fc1b9d8
1029 08:52:13.258322 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.258401 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136137}
W20251029 08:52:13.258713 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e8c4d793f10147fba5e4f44474575bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8c4d793f10147fba5e4f44474575bf0 already in progress: creating tablet
I20251029 08:52:13.258857 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 487) took 2134 ms. Trace:
W20251029 08:52:13.258951 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e3c24c0d32f74435bd9d28f2415e2c47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c24c0d32f74435bd9d28f2415e2c47 already in progress: creating tablet
I20251029 08:52:13.258953 1175 rpcz_store.cc:276] 1029 08:52:11.124606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.125328 (+ 722us) service_pool.cc:225] Handling call
1029 08:52:13.258740 (+2133412us) ts_tablet_manager.cc:1151] Deleting tablet 19fdc381a0d949b0b93cef399daa3021
1029 08:52:13.258749 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.258829 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132234}
I20251029 08:52:13.259186 1175 tablet_replica.cc:333] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.259392 1175 raft_consensus.cc:2243] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.259559 1175 raft_consensus.cc:2272] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.260161 1175 ts_tablet_manager.cc:1916] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.260613 888 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:13.263624 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet af38662420d14ac1a7cb5fd03fc1b9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af38662420d14ac1a7cb5fd03fc1b9d8 already in progress: creating tablet
W20251029 08:52:13.263922 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 19fdc381a0d949b0b93cef399daa3021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fdc381a0d949b0b93cef399daa3021 already in progress: creating tablet
I20251029 08:52:13.263819 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_081776ef8de44db6bec82f0f77a09aee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.264757 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:13.267998 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_081776ef8de44db6bec82f0f77a09aee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:13.268128 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 consensus state has no leader
I20251029 08:52:13.269531 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:13.270529 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5cfe4ea7c4e049d8af50f587af5f79f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.270254 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_87f770c9be434c09a9ddef0f8768c953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.271051 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cfe4ea7c4e049d8af50f587af5f79f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:13.271282 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87f770c9be434c09a9ddef0f8768c953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.273528 1175 ts_tablet_manager.cc:1929] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.273682 1175 log.cc:1199] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:13.274061 1175 ts_tablet_manager.cc:1950] T 09759a50a82f4b758449da9fbbc1f72d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.274858 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 488) took 2119 ms. Trace:
I20251029 08:52:13.274963 1175 rpcz_store.cc:276] 1029 08:52:11.156097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.156478 (+ 381us) service_pool.cc:225] Handling call
1029 08:52:13.259132 (+2102654us) ts_tablet_manager.cc:1151] Deleting tablet 09759a50a82f4b758449da9fbbc1f72d
1029 08:52:13.259140 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.263415 (+ 4275us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.273388 (+ 9973us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.274821 (+ 1433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101525}
I20251029 08:52:13.275399 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 09759a50a82f4b758449da9fbbc1f72d (table default.Table_0f93256544fb4b56adbbcaf8c50ecb42 [id=dc9c30acd58449e4bcb9217b628b59dc]) successfully deleted
I20251029 08:52:13.277048 1175 tablet_replica.cc:333] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.277724 1177 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.277832 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b8f1283f5a9245a599e5576a1e84b4a5
I20251029 08:52:13.277951 1177 ts_tablet_manager.cc:1403] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.007s
I20251029 08:52:13.278576 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8cc80787c54a42c4af323289ffd882e9" } new_table_name: "default.Table_49c30f52e0aa44519111145ec3d20c12" modify_external_catalogs: true
I20251029 08:52:13.278713 888 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.279417 1177 raft_consensus.cc:359] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.279598 1177 raft_consensus.cc:385] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.279729 1177 raft_consensus.cc:740] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.280053 1177 consensus_queue.cc:260] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.281424 1177 ts_tablet_manager.cc:1434] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:13.284976 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5f2dd9d6d0854e03a8298625938065c5
I20251029 08:52:13.285818 593 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e9a1022c7f654dd380fe3ee06449ad33
I20251029 08:52:13.285979 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.288362 1177 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.291209 1177 tablet_bootstrap.cc:654] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.291445 1175 raft_consensus.cc:2243] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.292491 1175 raft_consensus.cc:2272] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.293049 1175 ts_tablet_manager.cc:1916] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.303383 1175 ts_tablet_manager.cc:1929] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.305222 1175 log.cc:1199] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:13.306463 1175 ts_tablet_manager.cc:1950] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.309476 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 490) took 2068 ms. Trace:
I20251029 08:52:13.309847 1175 rpcz_store.cc:276] 1029 08:52:11.240517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.241391 (+ 874us) service_pool.cc:225] Handling call
1029 08:52:13.276953 (+2035562us) ts_tablet_manager.cc:1151] Deleting tablet a57b1dbbdf854bc6a6e1b5dc88b63bff
1029 08:52:13.276965 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.296476 (+ 19511us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.303248 (+ 6772us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.309437 (+ 6189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035316}
I20251029 08:52:13.310752 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (table default.Table_15261ce913ae430d8ea68e18a42df64a [id=a2015498bb844e87af722a69bcbf616a]) successfully deleted
I20251029 08:52:13.311316 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 498) took 1964 ms. Trace:
I20251029 08:52:13.311825 1175 rpcz_store.cc:276] 1029 08:52:11.346537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.346595 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:13.311149 (+1964554us) ts_tablet_manager.cc:1151] Deleting tablet e0fa089fce564af4b03aa323cec6cbf6
1029 08:52:13.311161 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.311284 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964295}
W20251029 08:52:13.312712 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e0fa089fce564af4b03aa323cec6cbf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fa089fce564af4b03aa323cec6cbf6 already in progress: creating tablet
I20251029 08:52:13.313285 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 497) took 1967 ms. Trace:
I20251029 08:52:13.313803 1175 rpcz_store.cc:276] 1029 08:52:11.346170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.348312 (+ 2142us) service_pool.cc:225] Handling call
1029 08:52:13.313103 (+1964791us) ts_tablet_manager.cc:1151] Deleting tablet 840ff254244f43a18679232279e02d1f
1029 08:52:13.313116 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.313225 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964474}
W20251029 08:52:13.314723 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 840ff254244f43a18679232279e02d1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840ff254244f43a18679232279e02d1f already in progress: creating tablet
I20251029 08:52:13.315227 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 496) took 1969 ms. Trace:
I20251029 08:52:13.315863 1175 rpcz_store.cc:276] 1029 08:52:11.345854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.356079 (+ 10225us) service_pool.cc:225] Handling call
1029 08:52:13.315087 (+1959008us) ts_tablet_manager.cc:1151] Deleting tablet be706771a95e45e5a20a15614eadfe1a
1029 08:52:13.315098 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.315194 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958698}
W20251029 08:52:13.316632 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet be706771a95e45e5a20a15614eadfe1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be706771a95e45e5a20a15614eadfe1a already in progress: creating tablet
I20251029 08:52:13.317131 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 510) took 1883 ms. Trace:
I20251029 08:52:13.317586 1175 rpcz_store.cc:276] 1029 08:52:11.433202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.433964 (+ 762us) service_pool.cc:225] Handling call
1029 08:52:13.316985 (+1883021us) ts_tablet_manager.cc:1151] Deleting tablet 608b7095d42941ccb0c4eb57f6754e74
1029 08:52:13.316995 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.317100 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882793}
W20251029 08:52:13.318346 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 608b7095d42941ccb0c4eb57f6754e74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608b7095d42941ccb0c4eb57f6754e74 already in progress: creating tablet
I20251029 08:52:13.318897 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 511) took 1885 ms. Trace:
I20251029 08:52:13.319344 1175 rpcz_store.cc:276] 1029 08:52:11.433336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.434227 (+ 891us) service_pool.cc:225] Handling call
1029 08:52:13.318760 (+1884533us) ts_tablet_manager.cc:1151] Deleting tablet a31847928db44bd4b1ebd50c5130065f
1029 08:52:13.318771 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.318866 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884330}
W20251029 08:52:13.320144 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a31847928db44bd4b1ebd50c5130065f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31847928db44bd4b1ebd50c5130065f already in progress: creating tablet
I20251029 08:52:13.320640 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 512) took 1884 ms. Trace:
I20251029 08:52:13.321106 1175 rpcz_store.cc:276] 1029 08:52:11.436198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.436296 (+ 98us) service_pool.cc:225] Handling call
1029 08:52:13.320491 (+1884195us) ts_tablet_manager.cc:1151] Deleting tablet 627bc8ff99394fa2abcca15d62e5ea10
1029 08:52:13.320502 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.320607 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863568}
W20251029 08:52:13.321810 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 627bc8ff99394fa2abcca15d62e5ea10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 627bc8ff99394fa2abcca15d62e5ea10 already in progress: creating tablet
I20251029 08:52:13.322204 1175 tablet_replica.cc:333] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.322665 1175 raft_consensus.cc:2243] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.323176 1175 raft_consensus.cc:2272] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.324869 1175 ts_tablet_manager.cc:1916] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.336933 1778 raft_consensus.cc:493] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.337136 1778 raft_consensus.cc:515] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.337929 1778 leader_election.cc:290] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:13.341145 1175 ts_tablet_manager.cc:1929] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.341807 1175 log.cc:1199] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/64f73597d1764540bcaf472d3e890ee8
I20251029 08:52:13.342826 1175 ts_tablet_manager.cc:1950] T 64f73597d1764540bcaf472d3e890ee8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.342821 1784 tablet.cc:1717] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:13.344399 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 513) took 1851 ms. Trace:
I20251029 08:52:13.344694 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.345800 1175 rpcz_store.cc:276] 1029 08:52:11.492706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.492773 (+ 67us) service_pool.cc:225] Handling call
1029 08:52:13.322139 (+1829366us) ts_tablet_manager.cc:1151] Deleting tablet 64f73597d1764540bcaf472d3e890ee8
1029 08:52:13.322149 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.328793 (+ 6644us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.341004 (+ 12211us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.344337 (+ 3333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829004}
I20251029 08:52:13.345903 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:13.346118 780 raft_consensus.cc:2468] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:13.346377 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 64f73597d1764540bcaf472d3e890ee8 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
I20251029 08:52:13.346688 861 leader_election.cc:304] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:13.346990 1778 raft_consensus.cc:2804] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:13.347074 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.347110 1778 raft_consensus.cc:493] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:13.347206 1778 raft_consensus.cc:3060] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:13.348167 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7046bb1fcdd34d308aecddfdf0752911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7046bb1fcdd34d308aecddfdf0752911 already in progress: creating tablet
I20251029 08:52:13.348708 1176 tablet_replica.cc:333] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:13.348917 1176 raft_consensus.cc:2243] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.349601 1176 raft_consensus.cc:2272] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.350183 1176 ts_tablet_manager.cc:1916] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.350303 1778 raft_consensus.cc:515] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.351024 1778 leader_election.cc:290] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:13.353375 761 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.353907 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:13.354082 780 raft_consensus.cc:3060] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:13.361502 780 raft_consensus.cc:2468] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:13.362118 861 leader_election.cc:304] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:13.364146 1779 raft_consensus.cc:2804] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:13.364372 1779 raft_consensus.cc:697] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:13.364724 1779 consensus_queue.cc:237] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.372386 1176 ts_tablet_manager.cc:1929] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:13.372633 1176 log.cc:1199] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/5f2dd9d6d0854e03a8298625938065c5
I20251029 08:52:13.373296 1176 ts_tablet_manager.cc:1950] T 5f2dd9d6d0854e03a8298625938065c5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:13.373692 1793 raft_consensus.cc:493] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.393638 1793 raft_consensus.cc:515] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.395035 1793 leader_election.cc:290] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:12.936374 1178 consensus_queue.cc:260] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.395429 1774 raft_consensus.cc:493] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.395572 1774 raft_consensus.cc:515] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.396485 1774 leader_election.cc:290] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.397135 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 5f2dd9d6d0854e03a8298625938065c5 (table default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c]) successfully deleted
W20251029 08:52:13.397565 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b8f1283f5a9245a599e5576a1e84b4a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f1283f5a9245a599e5576a1e84b4a5 already in progress: creating tablet
I20251029 08:52:13.398470 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.399413 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8daa66608cf94913afb94ef661b3c898" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.396615 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 519) took 1853 ms. Trace:
W20251029 08:52:13.400056 718 leader_election.cc:343] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.400218 1175 rpcz_store.cc:276] 1029 08:52:11.542624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.542945 (+ 321us) service_pool.cc:225] Handling call
1029 08:52:13.396398 (+1853453us) ts_tablet_manager.cc:1151] Deleting tablet 757e34137e884d5ca5cd88fdd03244d8
1029 08:52:13.396411 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.396541 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852106}
W20251029 08:52:13.400836 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 757e34137e884d5ca5cd88fdd03244d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757e34137e884d5ca5cd88fdd03244d8 already in progress: creating tablet
I20251029 08:52:13.401309 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 518) took 1859 ms. Trace:
I20251029 08:52:13.401450 1175 rpcz_store.cc:276] 1029 08:52:11.542325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.542398 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:13.401148 (+1858750us) ts_tablet_manager.cc:1151] Deleting tablet d5bb0fd4c62d47a2b10719ebd55851c0
1029 08:52:13.401159 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.401256 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856790}
W20251029 08:52:13.401962 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d5bb0fd4c62d47a2b10719ebd55851c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bb0fd4c62d47a2b10719ebd55851c0 already in progress: creating tablet
I20251029 08:52:13.402401 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 520) took 1834 ms. Trace:
I20251029 08:52:13.402534 1175 rpcz_store.cc:276] 1029 08:52:11.568144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.568234 (+ 90us) service_pool.cc:225] Handling call
1029 08:52:13.402251 (+1834017us) ts_tablet_manager.cc:1151] Deleting tablet 49d9c29b93a643ec837169a20c59de33
1029 08:52:13.402261 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.402346 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1833727}
I20251029 08:52:13.402710 1178 ts_tablet_manager.cc:1434] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.490s user 0.000s sys 0.003s
W20251029 08:52:13.403028 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 49d9c29b93a643ec837169a20c59de33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9c29b93a643ec837169a20c59de33 already in progress: creating tablet
I20251029 08:52:13.403443 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 521) took 1819 ms. Trace:
I20251029 08:52:13.403587 1175 rpcz_store.cc:276] 1029 08:52:11.584337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.596207 (+ 11870us) service_pool.cc:225] Handling call
1029 08:52:13.403301 (+1807094us) ts_tablet_manager.cc:1151] Deleting tablet 652903cbbdb042d780e7d71fc2c0ab4d
1029 08:52:13.403310 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.403392 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746198}
W20251029 08:52:13.404534 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 652903cbbdb042d780e7d71fc2c0ab4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652903cbbdb042d780e7d71fc2c0ab4d already in progress: creating tablet
I20251029 08:52:13.405045 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 524) took 1744 ms. Trace:
I20251029 08:52:13.408098 1175 rpcz_store.cc:276] 1029 08:52:11.660330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.660384 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:13.404888 (+1744504us) ts_tablet_manager.cc:1151] Deleting tablet fc04616301354cfe873c825a9f5d7ab6
1029 08:52:13.404898 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.405002 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1744216}
W20251029 08:52:13.408842 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fc04616301354cfe873c825a9f5d7ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc04616301354cfe873c825a9f5d7ab6 already in progress: creating tablet
I20251029 08:52:13.409772 1178 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:13.411940 1178 tablet_bootstrap.cc:654] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.413846 1175 tablet_replica.cc:333] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.414074 1175 raft_consensus.cc:2243] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.414294 1175 raft_consensus.cc:2272] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.414881 1175 ts_tablet_manager.cc:1916] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.417647 1751 raft_consensus.cc:2804] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 08:52:13.418391 998 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:13.422641 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_67e2e2e244b947f9b3b19e4c352c98e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.423305 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67e2e2e244b947f9b3b19e4c352c98e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:13.424539 861 consensus_peers.cc:597] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f2dd9d6d0854e03a8298625938065c5. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:13.426386 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 9702a7d8b8304b49a250c129ecf28303 with delete_type TABLET_DATA_DELETED (replaced by tablet 7be4bcf9e3324bd884e946fae20648fd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.428318 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9fdee9ea81294ee1806bada6c7a10d38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.428915 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fdee9ea81294ee1806bada6c7a10d38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.430176 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2fa23a8b35304536899bbbc02a070691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.430667 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fa23a8b35304536899bbbc02a070691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.444108 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.444582 1176 tablet_replica.cc:333] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:13.444908 1176 raft_consensus.cc:2243] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.445128 1176 raft_consensus.cc:2272] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.445782 1176 ts_tablet_manager.cc:1916] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.447479 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:13.447467 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:13.448225 861 leader_election.cc:343] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.447440 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:13.448992 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e687f593eead488db4bb445a63beeec7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:13.449551 720 leader_election.cc:343] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.449700 720 leader_election.cc:304] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.450204 1800 raft_consensus.cc:2749] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 08:52:13.450671 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f2dd9d6d0854e03a8298625938065c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.451005 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.451220 1751 raft_consensus.cc:493] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:13.451413 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f73597d1764540bcaf472d3e890ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3295bb0b47714080a5c3f34049acd487) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.451575 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 490) took 1989 ms. Trace:
W20251029 08:52:13.451675 861 leader_election.cc:343] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.456185 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8daa66608cf94913afb94ef661b3c898" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:13.456980 720 leader_election.cc:343] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.457160 720 leader_election.cc:304] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:13.457418 720 raft_consensus.cc:2686] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:13.458010 1751 raft_consensus.cc:3060] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:13.460747 1175 ts_tablet_manager.cc:1929] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.461701 1751 raft_consensus.cc:515] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:13.463241 998 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:13.464195 762 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:13.464520 996 leader_election.cc:343] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:13.464634 996 leader_election.cc:304] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.465049 1741 raft_consensus.cc:2749] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.465260 1179 rpcz_store.cc:276] 1029 08:52:11.417515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.428339 (+ 10824us) service_pool.cc:225] Handling call
1029 08:52:13.251736 (+1823397us) ts_tablet_manager.cc:1151] Deleting tablet a31847928db44bd4b1ebd50c5130065f
1029 08:52:13.251746 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.407169 (+155423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1823210}
W20251029 08:52:13.465849 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a31847928db44bd4b1ebd50c5130065f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31847928db44bd4b1ebd50c5130065f already in progress: creating tablet
I20251029 08:52:13.466027 1175 log.cc:1199] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/2e96cbdc4b7845b3823617a8e9a8c8b2
I20251029 08:52:13.466387 1183 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:13.466655 1175 ts_tablet_manager.cc:1950] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.467628 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 531) took 1729 ms. Trace:
I20251029 08:52:13.469605 1175 rpcz_store.cc:276] 1029 08:52:11.738524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.740000 (+ 1476us) service_pool.cc:225] Handling call
1029 08:52:13.409187 (+1669187us) ts_tablet_manager.cc:1151] Deleting tablet 2e96cbdc4b7845b3823617a8e9a8c8b2
1029 08:52:13.409197 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.424805 (+ 15608us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.460589 (+ 35784us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.467590 (+ 7001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665756}
I20251029 08:52:13.472365 1751 leader_election.cc:290] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:13.472844 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24ab9df6f768410eb4f66cffdcd68df2 consensus state has no leader
W20251029 08:52:13.474200 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47470a907c1949989f79e8a2b803dc25 consensus state has no leader
I20251029 08:52:13.474490 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b5051ff776e4975b917db0393a1f945 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:13.474644 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 9e1c023b679b4c3298bf953b53d4eb96 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:13.475616 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 79a3ef8389944c32bba7b3e598824118 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:13.476125 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a87e1550c7fe48a4ba7b9dbbbccdcf1a" } new_table_name: "default.Table_642c013e1f074c5c983b2df65a41dcb3" modify_external_catalogs: true
I20251029 08:52:13.476352 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 532) took 1702 ms. Trace:
I20251029 08:52:13.476372 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (table default.Table_b9b1c4a272214856ab50337015cdbd7b [id=4856bf7f477c47d58060d1728c544aa0]) successfully deleted
I20251029 08:52:13.476469 1175 rpcz_store.cc:276] 1029 08:52:11.773610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.784171 (+ 10561us) service_pool.cc:225] Handling call
1029 08:52:13.476175 (+1692004us) ts_tablet_manager.cc:1151] Deleting tablet f5b52314139945dd9c2e6ec899bf06dc
1029 08:52:13.476186 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.476317 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691701}
I20251029 08:52:13.477877 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 535) took 1661 ms. Trace:
W20251029 08:52:13.477969 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f5b52314139945dd9c2e6ec899bf06dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b52314139945dd9c2e6ec899bf06dc already in progress: creating tablet
I20251029 08:52:13.478350 1175 rpcz_store.cc:276] 1029 08:52:11.816567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.816621 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:13.477739 (+1661118us) ts_tablet_manager.cc:1151] Deleting tablet 633ace1413464e939e4e47782370e281
1029 08:52:13.477748 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.477829 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660900}
I20251029 08:52:13.479252 1175 tablet_replica.cc:333] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.479472 1175 raft_consensus.cc:2243] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.479657 1175 raft_consensus.cc:2272] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.480229 1175 ts_tablet_manager.cc:1916] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:13.480261 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 633ace1413464e939e4e47782370e281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 633ace1413464e939e4e47782370e281 already in progress: creating tablet
I20251029 08:52:13.480540 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_892ad3428acd46749ff80438c5b8f0d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.481122 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_892ad3428acd46749ff80438c5b8f0d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.483114 594 master_service.cc:954] ReplaceTablet: received request to replace tablet b8f2b34642704cd6bb966b6dcd494d51 from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:13.483703 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36c42e3bf3814769845a0204a00cf45f has no consensus state
I20251029 08:52:13.485538 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_41413dda9f934f0c8b39fb413de549b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.486061 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41413dda9f934f0c8b39fb413de549b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.489319 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_acc324f38e8943f4995a5f0bf3be0280" } modify_external_catalogs: true
I20251029 08:52:13.489487 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_acc324f38e8943f4995a5f0bf3be0280" } modify_external_catalogs: true
I20251029 08:52:13.491820 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0cde7b2972814447a6da2ec23d3b645f" } modify_external_catalogs: true
I20251029 08:52:13.492341 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f1d87583fc3b4975b8dc200489ff22ce" } new_table_name: "default.Table_79da38cb82164ec5acce7f9f37123031" modify_external_catalogs: true
I20251029 08:52:13.492910 1759 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:52:13.493013 1759 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: loading cluster ID for follower catalog manager: success
I20251029 08:52:13.493157 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0cde7b2972814447a6da2ec23d3b645f" } modify_external_catalogs: true
W20251029 08:52:13.496001 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a8c25ba4e9c410fa6d12949d6e595d7 consensus state has no leader
I20251029 08:52:13.500092 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2576901b40a44a1a892f8bf476ee9519" } modify_external_catalogs: true
I20251029 08:52:13.500294 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2576901b40a44a1a892f8bf476ee9519" } modify_external_catalogs: true
I20251029 08:52:13.500705 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2fc69b9cc1354fc1aba2e2bccac2e095" } modify_external_catalogs: true
I20251029 08:52:13.500850 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2fc69b9cc1354fc1aba2e2bccac2e095" } modify_external_catalogs: true
I20251029 08:52:13.501437 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b864bd5ff3e341938015d4bae27df6d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.501578 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d6dc6ba680874a51afe94c07adb83e2b" } modify_external_catalogs: true
I20251029 08:52:13.501704 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d6dc6ba680874a51afe94c07adb83e2b" } modify_external_catalogs: true
W20251029 08:52:13.502044 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b864bd5ff3e341938015d4bae27df6d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.503685 588 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8cec6c76b3c146ef969e9728afe8d2c7
I20251029 08:52:13.504112 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_655b57319ec4484288b2c34c169145ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.504638 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655b57319ec4484288b2c34c169145ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.511363 1803 tablet.cc:1722] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:13.511662 1175 ts_tablet_manager.cc:1929] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.511848 1175 log.cc:1199] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7792ee223dbe43aa82990541caced045
I20251029 08:52:13.512255 1175 ts_tablet_manager.cc:1950] T 7792ee223dbe43aa82990541caced045 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.513157 592 catalog_manager.cc:5649] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:13.514108 592 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:13.515079 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 939d7f6b895d460296eb132087ea65af (DEFAULT_TABLE table=default.Table_7f4559dca42a45bd9d4f2e050a759faa [id=6b34c4f10fcb4181841bdf1f85667414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.517084 888 tablet_service.cc:1505] Processing CreateTablet for tablet 939d7f6b895d460296eb132087ea65af (DEFAULT_TABLE table=default.Table_7f4559dca42a45bd9d4f2e050a759faa [id=6b34c4f10fcb4181841bdf1f85667414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.517587 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939d7f6b895d460296eb132087ea65af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.518522 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 536) took 1699 ms. Trace:
I20251029 08:52:13.518656 1175 rpcz_store.cc:276] 1029 08:52:11.818895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.819090 (+ 195us) service_pool.cc:225] Handling call
1029 08:52:13.479194 (+1660104us) ts_tablet_manager.cc:1151] Deleting tablet 7792ee223dbe43aa82990541caced045
1029 08:52:13.479202 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.497608 (+ 18406us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.511497 (+ 13889us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.518453 (+ 6956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659885}
I20251029 08:52:13.520473 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7792ee223dbe43aa82990541caced045 (table default.Table_9b80aae43bb845d1bb98687e0e6116f6 [id=335946e15b7c424ba3b769c46bd76ee9]) successfully deleted
I20251029 08:52:13.520967 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 537) took 1532 ms. Trace:
I20251029 08:52:13.521075 1175 rpcz_store.cc:276] 1029 08:52:11.988670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.991128 (+ 2458us) service_pool.cc:225] Handling call
1029 08:52:13.520769 (+1529641us) ts_tablet_manager.cc:1151] Deleting tablet b1aaf233eba2461f9182e094a0115a4e
1029 08:52:13.520781 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.520903 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529416}
W20251029 08:52:13.521559 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b1aaf233eba2461f9182e094a0115a4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1aaf233eba2461f9182e094a0115a4e already in progress: creating tablet
I20251029 08:52:13.524956 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 538) took 1535 ms. Trace:
I20251029 08:52:13.525077 1175 rpcz_store.cc:276] 1029 08:52:11.989002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.991418 (+ 2416us) service_pool.cc:225] Handling call
1029 08:52:13.524756 (+1533338us) ts_tablet_manager.cc:1151] Deleting tablet eeabf8f23751447aa9af02363c40361d
1029 08:52:13.524768 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.524901 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533180}
W20251029 08:52:13.525612 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet eeabf8f23751447aa9af02363c40361d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeabf8f23751447aa9af02363c40361d already in progress: creating tablet
I20251029 08:52:13.528491 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 533) took 1754 ms. Trace:
I20251029 08:52:13.529297 1175 rpcz_store.cc:276] 1029 08:52:11.773766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.828089 (+ 54323us) service_pool.cc:225] Handling call
1029 08:52:13.528304 (+1700215us) ts_tablet_manager.cc:1151] Deleting tablet 3e65b3b1c8d14743aca1465d95ace95f
1029 08:52:13.528315 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.528427 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533984}
W20251029 08:52:13.529817 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3e65b3b1c8d14743aca1465d95ace95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e65b3b1c8d14743aca1465d95ace95f already in progress: creating tablet
I20251029 08:52:13.529166 1177 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.530193 1177 ts_tablet_manager.cc:1403] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.242s user 0.032s sys 0.018s
I20251029 08:52:13.531068 1177 raft_consensus.cc:359] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.531332 1177 raft_consensus.cc:385] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.532807 1177 raft_consensus.cc:740] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.533118 1177 consensus_queue.cc:260] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.536444 751 tablet_service.cc:1505] Processing CreateTablet for tablet 939d7f6b895d460296eb132087ea65af (DEFAULT_TABLE table=default.Table_7f4559dca42a45bd9d4f2e050a759faa [id=6b34c4f10fcb4181841bdf1f85667414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.536967 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939d7f6b895d460296eb132087ea65af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.540490 1177 ts_tablet_manager.cc:1434] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:13.541563 1037 tablet_service.cc:1505] Processing CreateTablet for tablet fb22af7515084e099b41f9c1c92c067a (DEFAULT_TABLE table=default.Table_9357468257554705827bd3bc1eee7bbc [id=1ba24efae951426a8b0fd1c0766657e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.543012 762 tablet_service.cc:1505] Processing CreateTablet for tablet fb22af7515084e099b41f9c1c92c067a (DEFAULT_TABLE table=default.Table_9357468257554705827bd3bc1eee7bbc [id=1ba24efae951426a8b0fd1c0766657e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.543744 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22af7515084e099b41f9c1c92c067a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.544355 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 36c42e3bf3814769845a0204a00cf45f (DEFAULT_TABLE table=default.Table_49c30f52e0aa44519111145ec3d20c12 [id=babf57f5aba04a069a2a80d3324c6520]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.544524 896 tablet_service.cc:1505] Processing CreateTablet for tablet fb22af7515084e099b41f9c1c92c067a (DEFAULT_TABLE table=default.Table_9357468257554705827bd3bc1eee7bbc [id=1ba24efae951426a8b0fd1c0766657e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.544929 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22af7515084e099b41f9c1c92c067a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.545430 763 tablet_service.cc:1505] Processing CreateTablet for tablet 36c42e3bf3814769845a0204a00cf45f (DEFAULT_TABLE table=default.Table_49c30f52e0aa44519111145ec3d20c12 [id=babf57f5aba04a069a2a80d3324c6520]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.547307 1049 tablet_service.cc:1505] Processing CreateTablet for tablet b526d8c469e74e8fa4aa575e2af46767 (DEFAULT_TABLE table=default.Table_5ae1f06e4289499eb646135fd2430a0a [id=9c46d06c70f845ca8a259a83d87795e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.548343 899 tablet_service.cc:1505] Processing CreateTablet for tablet 36c42e3bf3814769845a0204a00cf45f (DEFAULT_TABLE table=default.Table_49c30f52e0aa44519111145ec3d20c12 [id=babf57f5aba04a069a2a80d3324c6520]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.548731 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c42e3bf3814769845a0204a00cf45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.548743 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c42e3bf3814769845a0204a00cf45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.549671 902 tablet_service.cc:1505] Processing CreateTablet for tablet b526d8c469e74e8fa4aa575e2af46767 (DEFAULT_TABLE table=default.Table_5ae1f06e4289499eb646135fd2430a0a [id=9c46d06c70f845ca8a259a83d87795e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.550045 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b526d8c469e74e8fa4aa575e2af46767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.553290 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d646f8ed2ef94c5da068e5f1fefd281e (DEFAULT_TABLE table=default.Table_bee86c1895544f8a9f972f5cc3662a3e [id=9dd4d64c20c3463fb4b20cf40a2a90ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:13.555672 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10baa0547180493f872a23614cf4c608 consensus state has no leader
I20251029 08:52:13.555970 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 07906300f1d54d4e8fde5165ee14c44f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:13.557464 756 tablet_service.cc:1505] Processing CreateTablet for tablet b526d8c469e74e8fa4aa575e2af46767 (DEFAULT_TABLE table=default.Table_5ae1f06e4289499eb646135fd2430a0a [id=9c46d06c70f845ca8a259a83d87795e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.557911 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b526d8c469e74e8fa4aa575e2af46767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.564625 1177 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:13.565798 996 consensus_peers.cc:597] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cf906c73dc84403bcc3dece74a822bd. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:13.566331 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 534) took 1761 ms. Trace:
I20251029 08:52:13.566510 1175 rpcz_store.cc:276] 1029 08:52:11.805011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.828649 (+ 23638us) service_pool.cc:225] Handling call
1029 08:52:13.566163 (+1737514us) ts_tablet_manager.cc:1151] Deleting tablet c9bbf6401423411089c9070f3b03f1a7
1029 08:52:13.566173 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.566275 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567874}
W20251029 08:52:13.567236 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c9bbf6401423411089c9070f3b03f1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bbf6401423411089c9070f3b03f1a7 already in progress: creating tablet
I20251029 08:52:13.567603 1175 tablet_replica.cc:333] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.567795 1175 raft_consensus.cc:2243] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.567847 1178 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:13.567996 1175 raft_consensus.cc:2272] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.568578 1175 ts_tablet_manager.cc:1916] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.568586 1178 ts_tablet_manager.cc:1403] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.005s
I20251029 08:52:13.569489 1178 raft_consensus.cc:359] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.569754 1178 raft_consensus.cc:385] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.569907 1178 raft_consensus.cc:740] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.570279 1178 consensus_queue.cc:260] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.571588 1177 tablet_bootstrap.cc:654] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.575040 1178 ts_tablet_manager.cc:1434] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:13.575516 1178 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:13.586892 1175 ts_tablet_manager.cc:1929] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.587090 1175 log.cc:1199] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:13.587253 1178 tablet_bootstrap.cc:654] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.587548 1175 ts_tablet_manager.cc:1950] T 4deeec492a2545ca86583c6ea899a0ab P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.591172 1179 tablet_replica.cc:333] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:13.591879 1179 raft_consensus.cc:2243] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:13.592658 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 542) took 1504 ms. Trace:
I20251029 08:52:13.602454 1179 pending_rounds.cc:70] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:13.602820 1179 pending_rounds.cc:77] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037612414263296 op_type: NO_OP noop_request { }
I20251029 08:52:13.603188 1179 raft_consensus.cc:2889] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:13.603523 1179 raft_consensus.cc:2272] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.605593 1179 ts_tablet_manager.cc:1916] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.606264 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939d7f6b895d460296eb132087ea65af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.607354 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b526d8c469e74e8fa4aa575e2af46767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.608433 1759 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: acquiring CA information for follower catalog manager: success
I20251029 08:52:13.612854 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d646f8ed2ef94c5da068e5f1fefd281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.613580 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c42e3bf3814769845a0204a00cf45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.614295 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22af7515084e099b41f9c1c92c067a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.616587 1178 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:13.616776 1178 ts_tablet_manager.cc:1403] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20251029 08:52:13.617645 1178 raft_consensus.cc:359] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.617846 1178 raft_consensus.cc:385] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.617902 1178 raft_consensus.cc:740] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.618196 1178 consensus_queue.cc:260] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.622443 1178 ts_tablet_manager.cc:1434] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:13.626932 1178 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:13.631996 998 consensus_peers.cc:597] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71ba5e1248dc4d4a8a4d7bb05b661a3f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:13.592846 1175 rpcz_store.cc:276] 1029 08:52:12.088560 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.088632 (+ 72us) service_pool.cc:225] Handling call
1029 08:52:13.567514 (+1478882us) ts_tablet_manager.cc:1151] Deleting tablet 4deeec492a2545ca86583c6ea899a0ab
1029 08:52:13.567522 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.579868 (+ 12346us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.586703 (+ 6835us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.592596 (+ 5893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473472}
I20251029 08:52:13.632687 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4deeec492a2545ca86583c6ea899a0ab (table default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e [id=58d71d681f8b4cd4967daf7d53eb5191]) successfully deleted
I20251029 08:52:13.633464 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 543) took 1497 ms. Trace:
I20251029 08:52:13.633714 1175 rpcz_store.cc:276] 1029 08:52:12.136303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.136381 (+ 78us) service_pool.cc:225] Handling call
1029 08:52:13.633176 (+1496795us) ts_tablet_manager.cc:1151] Deleting tablet a2b7853b85d24a04b8b11caaa4fe8c4d
1029 08:52:13.633197 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.633407 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496488}
W20251029 08:52:13.634462 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a2b7853b85d24a04b8b11caaa4fe8c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2b7853b85d24a04b8b11caaa4fe8c4d already in progress: creating tablet
I20251029 08:52:13.635087 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 544) took 1487 ms. Trace:
I20251029 08:52:13.637018 1175 rpcz_store.cc:276] 1029 08:52:12.147402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.147474 (+ 72us) service_pool.cc:225] Handling call
1029 08:52:13.634883 (+1487409us) ts_tablet_manager.cc:1151] Deleting tablet c7024e6eb4124109ad1575d216739fe2
1029 08:52:13.634899 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.635036 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477330}
W20251029 08:52:13.637763 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c7024e6eb4124109ad1575d216739fe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7024e6eb4124109ad1575d216739fe2 already in progress: creating tablet
I20251029 08:52:13.638393 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 546) took 1462 ms. Trace:
I20251029 08:52:13.638603 1175 rpcz_store.cc:276] 1029 08:52:12.176179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.178866 (+ 2687us) service_pool.cc:225] Handling call
1029 08:52:13.638150 (+1459284us) ts_tablet_manager.cc:1151] Deleting tablet 2ca466714c774fc9982ef145c6dd1571
1029 08:52:13.638167 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.638343 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438651}
W20251029 08:52:13.639290 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2ca466714c774fc9982ef145c6dd1571 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca466714c774fc9982ef145c6dd1571 already in progress: creating tablet
I20251029 08:52:13.639828 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 549) took 1416 ms. Trace:
I20251029 08:52:13.639994 1175 rpcz_store.cc:276] 1029 08:52:12.222894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.223631 (+ 737us) service_pool.cc:225] Handling call
1029 08:52:13.639647 (+1416016us) ts_tablet_manager.cc:1151] Deleting tablet 1dd62833c9d548adabbbf71856a4b181
1029 08:52:13.639662 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.639779 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403487}
W20251029 08:52:13.640972 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 1dd62833c9d548adabbbf71856a4b181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dd62833c9d548adabbbf71856a4b181 already in progress: creating tablet
I20251029 08:52:13.642038 1178 tablet_bootstrap.cc:654] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.642089 1779 raft_consensus.cc:493] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.642295 1779 raft_consensus.cc:515] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.643187 1779 leader_election.cc:290] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:13.641492 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 553) took 1227 ms. Trace:
W20251029 08:52:13.644757 861 leader_election.cc:343] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:13.645012 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55842c18dbef4513a61dd9629f2e5a4c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:13.645677 861 leader_election.cc:343] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.645845 861 leader_election.cc:304] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:13.646374 1779 raft_consensus.cc:2749] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.646737 1175 rpcz_store.cc:276] 1029 08:52:12.414207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.414366 (+ 159us) service_pool.cc:225] Handling call
1029 08:52:13.641317 (+1226951us) ts_tablet_manager.cc:1151] Deleting tablet e5fa181849474a559701fdb8c18d398b
1029 08:52:13.641330 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.641444 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226695}
W20251029 08:52:13.648139 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e5fa181849474a559701fdb8c18d398b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5fa181849474a559701fdb8c18d398b already in progress: creating tablet
I20251029 08:52:13.649202 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 554) took 1214 ms. Trace:
I20251029 08:52:13.652145 1175 rpcz_store.cc:276] 1029 08:52:12.434911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.449938 (+ 15027us) service_pool.cc:225] Handling call
1029 08:52:13.648973 (+1199035us) ts_tablet_manager.cc:1151] Deleting tablet 072cf27b0743488c8166e03b5e61c545
1029 08:52:13.648990 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.649154 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179258}
I20251029 08:52:13.653173 1800 raft_consensus.cc:493] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.653463 1800 raft_consensus.cc:515] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.654384 1800 leader_election.cc:290] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.656591 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:13.657276 720 leader_election.cc:343] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.655505 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.658497 1175 tablet_replica.cc:333] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.658821 1175 raft_consensus.cc:2243] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:13.659039 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 072cf27b0743488c8166e03b5e61c545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072cf27b0743488c8166e03b5e61c545 already in progress: creating tablet
I20251029 08:52:13.659282 1175 raft_consensus.cc:2272] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.660197 1175 ts_tablet_manager.cc:1916] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:13.660907 718 leader_election.cc:343] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.661088 718 leader_election.cc:304] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.661988 1774 raft_consensus.cc:2749] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:13.672683 996 consensus_peers.cc:597] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71ba5e1248dc4d4a8a4d7bb05b661a3f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:13.691098 1183 tablet_bootstrap.cc:654] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.697980 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8642fe3704384eaa803ad55dcdcdf6b6" } new_table_name: "default.Table_8dce1780dcf74468bd3a72090ccf984c" modify_external_catalogs: true
I20251029 08:52:13.699329 763 tablet_service.cc:1505] Processing CreateTablet for tablet d646f8ed2ef94c5da068e5f1fefd281e (DEFAULT_TABLE table=default.Table_bee86c1895544f8a9f972f5cc3662a3e [id=9dd4d64c20c3463fb4b20cf40a2a90ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.700690 902 tablet_service.cc:1505] Processing CreateTablet for tablet d646f8ed2ef94c5da068e5f1fefd281e (DEFAULT_TABLE table=default.Table_bee86c1895544f8a9f972f5cc3662a3e [id=9dd4d64c20c3463fb4b20cf40a2a90ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.701274 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d646f8ed2ef94c5da068e5f1fefd281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.704255 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d646f8ed2ef94c5da068e5f1fefd281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.705472 761 tablet_service.cc:1505] Processing CreateTablet for tablet 2eed71b7f27b4f88bdba422277dc3627 (DEFAULT_TABLE table=default.Table_ebfdc32c8a6f48359f16be99eaef1443 [id=8e36b33f0fcf433e804c9c7d4d1dae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.706542 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eed71b7f27b4f88bdba422277dc3627. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.716086 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.721453 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 2eed71b7f27b4f88bdba422277dc3627 (DEFAULT_TABLE table=default.Table_ebfdc32c8a6f48359f16be99eaef1443 [id=8e36b33f0fcf433e804c9c7d4d1dae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.727478 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:13.728152 590 catalog_manager.cc:6719] ReplaceTablet: tablet 9e1c023b679b4c3298bf953b53d4eb96 deleted and replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5
I20251029 08:52:13.730700 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6ba990fb812943a5bd7c7588dbb684dd" } new_table_name: "default.Table_438d2aad28fd458786870f8dd77671ec" modify_external_catalogs: true
I20251029 08:52:13.732033 1177 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.732192 1177 ts_tablet_manager.cc:1403] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.168s user 0.054s sys 0.075s
I20251029 08:52:13.732833 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.733039 1177 raft_consensus.cc:359] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.733232 1177 raft_consensus.cc:385] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.733373 1177 raft_consensus.cc:740] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.733645 1177 consensus_queue.cc:260] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.734362 1177 ts_tablet_manager.cc:1434] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:13.734790 1177 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.735814 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0a016861cf8d4f54a567239750fc81df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.736465 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a016861cf8d4f54a567239750fc81df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.736754 759 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.737165 1177 tablet_bootstrap.cc:654] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.737330 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.742100 1774 raft_consensus.cc:493] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.742264 1774 raft_consensus.cc:515] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.742939 1774 leader_election.cc:290] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.744962 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757e34137e884d5ca5cd88fdd03244d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:13.745766 718 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.749727 902 tablet_service.cc:1505] Processing CreateTablet for tablet 2eed71b7f27b4f88bdba422277dc3627 (DEFAULT_TABLE table=default.Table_ebfdc32c8a6f48359f16be99eaef1443 [id=8e36b33f0fcf433e804c9c7d4d1dae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.751964 1178 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:13.752218 1178 ts_tablet_manager.cc:1403] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.125s user 0.024s sys 0.020s
I20251029 08:52:13.752727 1177 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.752908 1177 ts_tablet_manager.cc:1403] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20251029 08:52:13.753728 1177 raft_consensus.cc:359] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.753942 1177 raft_consensus.cc:385] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.754020 1177 raft_consensus.cc:740] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.753839 1178 raft_consensus.cc:359] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.754225 1178 raft_consensus.cc:385] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.754282 1177 consensus_queue.cc:260] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.754362 1178 raft_consensus.cc:740] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.754799 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 107206cf33054e3fb880190eab8cd1cc
I20251029 08:52:13.754896 1177 ts_tablet_manager.cc:1434] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:13.755627 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ac5705ab49ff41f5ac459ed7fe561419"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.755817 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 107206cf33054e3fb880190eab8cd1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:13.756214 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac5705ab49ff41f5ac459ed7fe561419 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.759227 763 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa99eb90b0d48a9bcbc10c38209546c (DEFAULT_TABLE table=default.Table_95e4062d439d4ac48a856012207ec7f8 [id=0a8e1c0ccd1b46b4a4d91ed95954db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.759480 1178 consensus_queue.cc:260] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.759716 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa99eb90b0d48a9bcbc10c38209546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.760330 1177 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.761976 1177 tablet_bootstrap.cc:654] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.765391 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eed71b7f27b4f88bdba422277dc3627. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.765923 1178 ts_tablet_manager.cc:1434] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251029 08:52:13.767272 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 107206cf33054e3fb880190eab8cd1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.769814 1178 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:13.771730 1178 tablet_bootstrap.cc:654] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.772871 1774 raft_consensus.cc:493] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.773041 1774 raft_consensus.cc:515] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.773761 1774 leader_election.cc:290] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.776041 1759 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 08:52:13.781886 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757e34137e884d5ca5cd88fdd03244d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:13.783670 1175 ts_tablet_manager.cc:1929] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:13.783850 1177 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.784008 1177 ts_tablet_manager.cc:1403] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20251029 08:52:13.783888 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:13.784869 1177 raft_consensus.cc:359] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.785049 1177 raft_consensus.cc:385] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.785146 1177 raft_consensus.cc:740] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.785247 1175 log.cc:1199] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a9cf61d6137544019bca25acf2431986
I20251029 08:52:13.785413 1177 consensus_queue.cc:260] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.785918 1175 ts_tablet_manager.cc:1950] T a9cf61d6137544019bca25acf2431986 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:13.786337 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 560) took 1011 ms. Trace:
I20251029 08:52:13.786491 1175 rpcz_store.cc:276] 1029 08:52:12.774373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.774434 (+ 61us) service_pool.cc:225] Handling call
1029 08:52:13.658387 (+883953us) ts_tablet_manager.cc:1151] Deleting tablet a9cf61d6137544019bca25acf2431986
1029 08:52:13.658404 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.753504 (+ 95100us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.783555 (+ 30051us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.786275 (+ 2720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":883692}
I20251029 08:52:13.787070 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]) successfully deleted
I20251029 08:52:13.787808 1175 tablet_replica.cc:333] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:13.788085 1175 raft_consensus.cc:2243] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.789057 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eed71b7f27b4f88bdba422277dc3627. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.790043 1177 ts_tablet_manager.cc:1434] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 08:52:13.791673 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 107206cf33054e3fb880190eab8cd1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.792011 1177 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:13.792797 720 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.792946 720 leader_election.cc:304] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.793419 1774 raft_consensus.cc:2749] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:13.793819 718 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.796173 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:13.796792 720 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.796909 720 leader_election.cc:304] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.797302 1774 raft_consensus.cc:2749] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.798840 1177 tablet_bootstrap.cc:654] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.800810 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa99eb90b0d48a9bcbc10c38209546c (DEFAULT_TABLE table=default.Table_95e4062d439d4ac48a856012207ec7f8 [id=0a8e1c0ccd1b46b4a4d91ed95954db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.801266 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa99eb90b0d48a9bcbc10c38209546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.810106 1177 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.810277 1177 ts_tablet_manager.cc:1403] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20251029 08:52:13.810974 1177 raft_consensus.cc:359] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.811169 1177 raft_consensus.cc:385] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.811242 1177 raft_consensus.cc:740] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.811511 1177 consensus_queue.cc:260] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.815754 1177 ts_tablet_manager.cc:1434] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:13.816253 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a506e35afe6a4c87901f27c434f3b348
I20251029 08:52:13.817958 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a506e35afe6a4c87901f27c434f3b348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.818233 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a506e35afe6a4c87901f27c434f3b348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.818802 1177 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.820746 1177 tablet_bootstrap.cc:654] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.821571 902 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa99eb90b0d48a9bcbc10c38209546c (DEFAULT_TABLE table=default.Table_95e4062d439d4ac48a856012207ec7f8 [id=0a8e1c0ccd1b46b4a4d91ed95954db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.822019 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa99eb90b0d48a9bcbc10c38209546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.828812 1177 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.828989 1177 ts_tablet_manager.cc:1403] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251029 08:52:13.829840 1177 raft_consensus.cc:359] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.830050 1177 raft_consensus.cc:385] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.830158 1177 raft_consensus.cc:740] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.830448 1177 consensus_queue.cc:260] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.831869 1779 raft_consensus.cc:493] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.832010 1779 raft_consensus.cc:515] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:13.832798 1779 leader_election.cc:290] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:13.837425 861 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:13.838518 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_11375103cdf341e8a98dca7cca956197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.838930 1177 ts_tablet_manager.cc:1434] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20251029 08:52:13.839792 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11375103cdf341e8a98dca7cca956197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.842347 1177 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.844693 1177 tablet_bootstrap.cc:654] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.846603 1177 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.846755 1177 ts_tablet_manager.cc:1403] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251029 08:52:13.847507 1177 raft_consensus.cc:359] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.847725 1177 raft_consensus.cc:385] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.847806 1177 raft_consensus.cc:740] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.848086 1177 consensus_queue.cc:260] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.848608 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:13.850339 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.850745 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.851658 1177 ts_tablet_manager.cc:1434] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:13.853549 751 tablet_service.cc:1505] Processing CreateTablet for tablet e039e5f7f941410cb42a21502a1844a4 (DEFAULT_TABLE table=default.Table_c6c5865ac9e944ea97b59f1b9e274284 [id=bdc372216bd34d3a8ab3e9848f270b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.853988 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e039e5f7f941410cb42a21502a1844a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.854141 902 tablet_service.cc:1505] Processing CreateTablet for tablet e039e5f7f941410cb42a21502a1844a4 (DEFAULT_TABLE table=default.Table_c6c5865ac9e944ea97b59f1b9e274284 [id=bdc372216bd34d3a8ab3e9848f270b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.855240 761 tablet_service.cc:1505] Processing CreateTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.855640 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4190dc9e09b4480a79774d1e0b5bc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.855836 896 tablet_service.cc:1505] Processing CreateTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.857299 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4190dc9e09b4480a79774d1e0b5bc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.860332 1177 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.860538 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e039e5f7f941410cb42a21502a1844a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.862087 1177 tablet_bootstrap.cc:654] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:13.863306 1175 raft_consensus.cc:2272] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:13.863909 1175 ts_tablet_manager.cc:1916] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:13.866003 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet a506e35afe6a4c87901f27c434f3b348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.866219 1751 raft_consensus.cc:493] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:13.866386 1751 raft_consensus.cc:515] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.867199 1751 leader_election.cc:290] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:13.867501 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.868068 1051 tablet_service.cc:1505] Processing CreateTablet for tablet e039e5f7f941410cb42a21502a1844a4 (DEFAULT_TABLE table=default.Table_c6c5865ac9e944ea97b59f1b9e274284 [id=bdc372216bd34d3a8ab3e9848f270b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.868628 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e039e5f7f941410cb42a21502a1844a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.869305 1176 ts_tablet_manager.cc:1929] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:13.869448 1176 log.cc:1199] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8daa66608cf94913afb94ef661b3c898
I20251029 08:52:13.869807 1176 ts_tablet_manager.cc:1950] T 8daa66608cf94913afb94ef661b3c898 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:13.871158 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8daa66608cf94913afb94ef661b3c898 (table default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]) successfully deleted
I20251029 08:52:13.871229 1176 tablet_replica.cc:333] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:13.871420 1176 raft_consensus.cc:2243] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:13.871592 1176 raft_consensus.cc:2272] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:13.871659 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa0ef904c2fb4dfeadbf56ae659c4e20 already in progress: creating tablet
I20251029 08:52:13.872366 1033 tablet_service.cc:1505] Processing CreateTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.872879 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4190dc9e09b4480a79774d1e0b5bc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.876621 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.879256 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2858a0baf1f04df3864e662fe0d23366" } new_table_name: "default.Table_ee1b6420e8a74932826b5a476ee2d9b2" modify_external_catalogs: true
I20251029 08:52:13.884219 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b8f2b34642704cd6bb966b6dcd494d51
I20251029 08:52:13.884864 594 catalog_manager.cc:6719] ReplaceTablet: tablet b8f2b34642704cd6bb966b6dcd494d51 deleted and replaced by tablet 6f91eb858ae344549a78357e04f7e938
I20251029 08:52:13.885259 761 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:13.886237 899 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.893429 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dd8cd3b63a734120b7e5d8da5bcad3e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.894207 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd8cd3b63a734120b7e5d8da5bcad3e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.897404 1177 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:13.897595 1177 ts_tablet_manager.cc:1403] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20251029 08:52:13.898083 756 tablet_service.cc:1505] Processing CreateTablet for tablet 400c4991955b47e799df66c05caee6e3 (DEFAULT_TABLE table=default.Table_ff59f466b5804429bc09b24c36220b1f [id=00108aec8bd148fcb37aca3506875311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.898509 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400c4991955b47e799df66c05caee6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.898464 1177 raft_consensus.cc:359] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.898658 1177 raft_consensus.cc:385] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:13.898751 1177 raft_consensus.cc:740] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:13.900000 899 tablet_service.cc:1505] Processing CreateTablet for tablet 400c4991955b47e799df66c05caee6e3 (DEFAULT_TABLE table=default.Table_ff59f466b5804429bc09b24c36220b1f [id=00108aec8bd148fcb37aca3506875311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.900458 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400c4991955b47e799df66c05caee6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.899004 1177 consensus_queue.cc:260] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:13.912146 1177 ts_tablet_manager.cc:1434] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 08:52:13.912508 1177 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:13.915105 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39dc5ebeb36043e0aa5955bacd06c2ef" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:13.925913 1177 tablet_bootstrap.cc:654] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:13.929826 998 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 64f73597d1764540bcaf472d3e890ee8
W20251029 08:52:13.931703 998 consensus_peers.cc:597] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09759a50a82f4b758449da9fbbc1f72d. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:13.932075 861 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:13.932240 861 leader_election.cc:304] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:13.932684 1779 raft_consensus.cc:2749] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:13.933735 996 leader_election.cc:343] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 64f73597d1764540bcaf472d3e890ee8
I20251029 08:52:13.933889 996 leader_election.cc:304] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:13.934516 1751 raft_consensus.cc:2749] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:13.935793 1803 tablet.cc:1717] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Skipping requested alter to schema version 2, tablet already version 2
I20251029 08:52:13.940115 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:13.944479 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 48003.1us
I20251029 08:52:13.946628 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 400c4991955b47e799df66c05caee6e3 (DEFAULT_TABLE table=default.Table_ff59f466b5804429bc09b24c36220b1f [id=00108aec8bd148fcb37aca3506875311]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:13.951347 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 74fe974a368a4afe8a5da6dfd1036400 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 74fe974a368a4afe8a5da6dfd1036400 consensus state has no leader
I20251029 08:52:13.956629 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:13.963524 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 07906300f1d54d4e8fde5165ee14c44f
I20251029 08:52:13.964010 591 catalog_manager.cc:6719] ReplaceTablet: tablet 07906300f1d54d4e8fde5165ee14c44f deleted and replaced by tablet a92cc4a93a584dffaafe21316fe8dc06
I20251029 08:52:13.966269 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bc205fa7964840f9802b4bb2a9a4f954 [id=821160def5624241afdce0f734ef2411]). Expected version 2 got 0
I20251029 08:52:13.966574 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400c4991955b47e799df66c05caee6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.967072 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.967648 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]). Expected version 2 got 0
I20251029 08:52:13.967629 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 107206cf33054e3fb880190eab8cd1cc (DEFAULT_TABLE table=default.Table_2fc69b9cc1354fc1aba2e2bccac2e095 [id=fb68321c3728400da5d2339b86acd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.967824 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
I20251029 08:52:13.971846 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.971776 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_09655491a5f442aaa477ef4bbb1303c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:13.972133 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.972333 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:13.972409 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09655491a5f442aaa477ef4bbb1303c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.972522 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.973349 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.973665 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d089089a84af48068f5a61c67b1ea446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.974112 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d089089a84af48068f5a61c67b1ea446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.976889 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:13.977593 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c488d79d59554375912c487ac611bf0a" } new_table_name: "default.Table_d74312a6b49e490ab01040a51e1e75e7" modify_external_catalogs: true
I20251029 08:52:13.978821 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107206cf33054e3fb880190eab8cd1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.980393 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:13.985152 902 tablet_service.cc:1505] Processing CreateTablet for tablet 107206cf33054e3fb880190eab8cd1cc (DEFAULT_TABLE table=default.Table_2fc69b9cc1354fc1aba2e2bccac2e095 [id=fb68321c3728400da5d2339b86acd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.985939 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107206cf33054e3fb880190eab8cd1cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:13.990837 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1da3e7940352460c9b2bb060174829c9 consensus state has no leader
I20251029 08:52:13.993178 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_55b3b56ad90e4db1b7b8282560023ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:13.993808 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55b3b56ad90e4db1b7b8282560023ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:13.994475 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 116cc0d155e24653937953eea4f39ee4 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:13.995155 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 116cc0d155e24653937953eea4f39ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:13.999850 1175 ts_tablet_manager.cc:1929] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.004918 896 tablet_service.cc:1505] Processing CreateTablet for tablet 116cc0d155e24653937953eea4f39ee4 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.005343 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 116cc0d155e24653937953eea4f39ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.012395 1179 ts_tablet_manager.cc:1929] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:14.017056 1779 raft_consensus.cc:493] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.017221 1779 raft_consensus.cc:515] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.017907 1779 leader_election.cc:290] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.019145 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.019806 861 leader_election.cc:343] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.020067 1779 raft_consensus.cc:493] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.020211 1779 raft_consensus.cc:515] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.020748 1779 leader_election.cc:290] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:14.021528 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e67ddf5ceb148dfad8f7be9f26cff64" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.021970 1179 log.cc:1199] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4cf906c73dc84403bcc3dece74a822bd
W20251029 08:52:14.022215 861 leader_election.cc:343] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.022517 1179 ts_tablet_manager.cc:1950] T 4cf906c73dc84403bcc3dece74a822bd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.023356 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 492) took 2525 ms. Trace:
I20251029 08:52:14.023931 1179 rpcz_store.cc:276] 1029 08:52:11.497376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.497516 (+ 140us) service_pool.cc:225] Handling call
1029 08:52:13.469863 (+1972347us) ts_tablet_manager.cc:1151] Deleting tablet 4cf906c73dc84403bcc3dece74a822bd
1029 08:52:13.589361 (+119498us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.920496 (+331135us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.994969 (+ 74473us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.023278 (+ 28309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969217}
I20251029 08:52:14.024355 1177 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.024513 1177 ts_tablet_manager.cc:1403] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.000s
I20251029 08:52:14.024986 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4cf906c73dc84403bcc3dece74a822bd (table default.Table_bc1f6a8467ac4e1084471956749c7145 [id=b253b07dae944d078fe7589af7e3adfd]) successfully deleted
I20251029 08:52:14.025254 1177 raft_consensus.cc:359] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.025440 1177 raft_consensus.cc:385] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.025506 1177 raft_consensus.cc:740] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.025767 1177 consensus_queue.cc:260] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:14.027056 996 consensus_peers.cc:597] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09759a50a82f4b758449da9fbbc1f72d. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:14.027484 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 491) took 2534 ms. Trace:
I20251029 08:52:14.027655 1179 rpcz_store.cc:276] 1029 08:52:11.493012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.493164 (+ 152us) service_pool.cc:225] Handling call
1029 08:52:14.027261 (+2534097us) ts_tablet_manager.cc:1151] Deleting tablet 627bc8ff99394fa2abcca15d62e5ea10
1029 08:52:14.027276 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.027423 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524364}
W20251029 08:52:14.028326 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 627bc8ff99394fa2abcca15d62e5ea10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 627bc8ff99394fa2abcca15d62e5ea10 already in progress: creating tablet
I20251029 08:52:14.028891 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 497) took 2423 ms. Trace:
I20251029 08:52:14.029067 1179 rpcz_store.cc:276] 1029 08:52:11.605133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.605184 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:14.028691 (+2423507us) ts_tablet_manager.cc:1151] Deleting tablet d5bb0fd4c62d47a2b10719ebd55851c0
1029 08:52:14.028702 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.028831 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381690}
I20251029 08:52:14.032347 1177 ts_tablet_manager.cc:1434] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:14.036207 899 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.037950 1177 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.040316 1177 tablet_bootstrap.cc:654] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:14.043437 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07bf5d693be240f18bacccc81a6ea5d4 consensus state has no leader
I20251029 08:52:14.044754 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:14.045486 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2b2df448cb2e4ac89b131af3732c83fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.046052 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b2df448cb2e4ac89b131af3732c83fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.047513 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_208a79319e20472782aba43b998349ea" } modify_external_catalogs: true
I20251029 08:52:14.047699 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_208a79319e20472782aba43b998349ea" } modify_external_catalogs: true
W20251029 08:52:14.061941 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d5bb0fd4c62d47a2b10719ebd55851c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bb0fd4c62d47a2b10719ebd55851c0 already in progress: creating tablet
I20251029 08:52:14.061204 1183 tablet_bootstrap.cc:492] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.062427 1183 ts_tablet_manager.cc:1403] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.596s user 0.003s sys 0.005s
I20251029 08:52:14.064741 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 498) took 2458 ms. Trace:
I20251029 08:52:14.066556 1179 rpcz_store.cc:276] 1029 08:52:11.606301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.606349 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:14.064508 (+2458159us) ts_tablet_manager.cc:1151] Deleting tablet 757e34137e884d5ca5cd88fdd03244d8
1029 08:52:14.064523 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.064671 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417423}
I20251029 08:52:14.063549 1183 raft_consensus.cc:359] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.067459 1183 raft_consensus.cc:385] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.069314 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:14.070298 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5a8eb1fd4c9c425ca97d2fb315b7e555" } new_table_name: "default.Table_0b00cd7c457d4204965de864c88be5e6" modify_external_catalogs: true
I20251029 08:52:14.071333 1176 ts_tablet_manager.cc:1916] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:14.072696 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 757e34137e884d5ca5cd88fdd03244d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757e34137e884d5ca5cd88fdd03244d8 already in progress: creating tablet
I20251029 08:52:14.074023 593 master_service.cc:954] ReplaceTablet: received request to replace tablet e0fa089fce564af4b03aa323cec6cbf6 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.077456 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ec00cd64dc247da8442b6ba197f370a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.080080 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.080312 761 tablet_service.cc:1505] Processing CreateTablet for tablet 107206cf33054e3fb880190eab8cd1cc (DEFAULT_TABLE table=default.Table_2fc69b9cc1354fc1aba2e2bccac2e095 [id=fb68321c3728400da5d2339b86acd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.080499 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:14.080899 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.080935 782 raft_consensus.cc:2468] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:14.081306 861 leader_election.cc:304] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:14.081365 759 tablet_service.cc:1505] Processing CreateTablet for tablet 116cc0d155e24653937953eea4f39ee4 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.081782 1779 raft_consensus.cc:2804] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:14.081872 1779 raft_consensus.cc:493] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:14.081964 1779 raft_consensus.cc:3060] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.082212 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107206cf33054e3fb880190eab8cd1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.082134 1183 raft_consensus.cc:740] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.082656 1183 consensus_queue.cc:260] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.082010 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 500) took 2432 ms. Trace:
I20251029 08:52:14.082890 1179 rpcz_store.cc:276] 1029 08:52:11.649250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.649354 (+ 104us) service_pool.cc:225] Handling call
1029 08:52:14.081837 (+2432483us) ts_tablet_manager.cc:1151] Deleting tablet 652903cbbdb042d780e7d71fc2c0ab4d
1029 08:52:14.081848 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.081953 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432203}
I20251029 08:52:14.084949 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 116cc0d155e24653937953eea4f39ee4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.084964 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 652903cbbdb042d780e7d71fc2c0ab4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652903cbbdb042d780e7d71fc2c0ab4d already in progress: creating tablet
I20251029 08:52:14.085068 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 499) took 2449 ms. Trace:
I20251029 08:52:14.085147 1179 rpcz_store.cc:276] 1029 08:52:11.635988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.636070 (+ 82us) service_pool.cc:225] Handling call
1029 08:52:14.084899 (+2448829us) ts_tablet_manager.cc:1151] Deleting tablet 49d9c29b93a643ec837169a20c59de33
1029 08:52:14.084911 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.085022 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422888}
W20251029 08:52:14.085258 861 leader_election.cc:343] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:14.085393 861 leader_election.cc:304] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:14.085587 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 49d9c29b93a643ec837169a20c59de33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9c29b93a643ec837169a20c59de33 already in progress: creating tablet
I20251029 08:52:14.086566 1779 raft_consensus.cc:515] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.088099 1779 leader_election.cc:290] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.088383 1779 raft_consensus.cc:2749] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.088996 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 504) took 2334 ms. Trace:
I20251029 08:52:14.089082 1179 rpcz_store.cc:276] 1029 08:52:11.754471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.755447 (+ 976us) service_pool.cc:225] Handling call
1029 08:52:14.088847 (+2333400us) ts_tablet_manager.cc:1151] Deleting tablet fc04616301354cfe873c825a9f5d7ab6
1029 08:52:14.088857 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.088945 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2330467}
W20251029 08:52:14.089453 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fc04616301354cfe873c825a9f5d7ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc04616301354cfe873c825a9f5d7ab6 already in progress: creating tablet
I20251029 08:52:14.089860 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 512) took 2250 ms. Trace:
I20251029 08:52:14.090109 1179 rpcz_store.cc:276] 1029 08:52:11.838881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.838974 (+ 93us) service_pool.cc:225] Handling call
1029 08:52:14.089726 (+2250752us) ts_tablet_manager.cc:1151] Deleting tablet c9bbf6401423411089c9070f3b03f1a7
1029 08:52:14.089735 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.089814 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2244834}
W20251029 08:52:14.100847 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dec33b502aa4916b89b878462de1ab1 consensus state has no leader
I20251029 08:52:13.999990 1175 log.cc:1199] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/983a406aeed243f29ab322f94a0f5239
I20251029 08:52:14.102654 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4f3ecd8febb64821a555373de62d6b6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.102860 1175 ts_tablet_manager.cc:1950] T 983a406aeed243f29ab322f94a0f5239 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:14.103139 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f3ecd8febb64821a555373de62d6b6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.103217 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 559) took 1336 ms. Trace:
I20251029 08:52:14.103335 1175 rpcz_store.cc:276] 1029 08:52:12.766526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.768640 (+ 2114us) service_pool.cc:225] Handling call
1029 08:52:13.787706 (+1019066us) ts_tablet_manager.cc:1151] Deleting tablet 983a406aeed243f29ab322f94a0f5239
1029 08:52:13.787717 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:13.985562 (+197845us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:13.999710 (+ 14148us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.103174 (+103464us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74906,"tablet-delete.queue_time_us":1012889}
I20251029 08:52:14.103826 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 983a406aeed243f29ab322f94a0f5239 (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
I20251029 08:52:14.105522 1177 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.108366 1177 ts_tablet_manager.cc:1403] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20251029 08:52:14.112095 1177 raft_consensus.cc:359] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.114500 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2f33511f2bcc46a0a692fa6f7827a66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.114677 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_41413dda9f934f0c8b39fb413de549b8" } modify_external_catalogs: true
I20251029 08:52:14.114876 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_41413dda9f934f0c8b39fb413de549b8" } modify_external_catalogs: true
W20251029 08:52:14.115103 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f33511f2bcc46a0a692fa6f7827a66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.118564 1177 raft_consensus.cc:385] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.118680 1177 raft_consensus.cc:740] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.118948 1177 consensus_queue.cc:260] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.117810 1175 tablet_replica.cc:333] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.123242 1175 raft_consensus.cc:2243] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.123864 1175 raft_consensus.cc:2272] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.126494 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:14.128033 781 raft_consensus.cc:3060] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.128621 1176 ts_tablet_manager.cc:1929] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.132879 1176 log.cc:1199] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:14.102301 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:14.123660 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 7792ee223dbe43aa82990541caced045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.124205 1183 ts_tablet_manager.cc:1434] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
W20251029 08:52:14.135087 861 leader_election.cc:343] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.136528 781 raft_consensus.cc:2468] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:14.137321 861 leader_election.cc:304] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:14.137734 1820 raft_consensus.cc:2804] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:14.137902 1820 raft_consensus.cc:697] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:14.138187 1820 consensus_queue.cc:237] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.139038 1183 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.140093 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cd2e9a95b6c40be9e17397a313d249b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.142155 1183 tablet_bootstrap.cc:654] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.142313 1176 ts_tablet_manager.cc:1950] T 757e34137e884d5ca5cd88fdd03244d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:14.142624 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c9bbf6401423411089c9070f3b03f1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bbf6401423411089c9070f3b03f1a7 already in progress: creating tablet
I20251029 08:52:14.143344 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 757e34137e884d5ca5cd88fdd03244d8 (table default.Table_4a022119f31d4460af5cdd68baa8521b [id=9eab4fb7fff44c10854a84cbf994f667]) successfully deleted
I20251029 08:52:14.143415 1779 raft_consensus.cc:493] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.143800 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 510) took 2305 ms. Trace:
I20251029 08:52:14.148811 1176 tablet_replica.cc:333] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.148778 1179 rpcz_store.cc:276] 1029 08:52:11.838677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.839140 (+ 463us) service_pool.cc:225] Handling call
1029 08:52:14.143146 (+2304006us) ts_tablet_manager.cc:1151] Deleting tablet 3e65b3b1c8d14743aca1465d95ace95f
1029 08:52:14.143157 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.143749 (+ 592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2298148}
I20251029 08:52:14.148991 1176 raft_consensus.cc:2243] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.149143 1176 raft_consensus.cc:2272] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.149516 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3e65b3b1c8d14743aca1465d95ace95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e65b3b1c8d14743aca1465d95ace95f already in progress: creating tablet
I20251029 08:52:14.149711 1176 ts_tablet_manager.cc:1916] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.150271 1179 tablet_replica.cc:333] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.150516 1179 raft_consensus.cc:2243] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:14.150874 1179 pending_rounds.cc:70] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:14.150985 1179 pending_rounds.cc:77] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037611755237376 op_type: NO_OP noop_request { }
I20251029 08:52:14.151232 1179 raft_consensus.cc:2889] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:14.151393 1179 raft_consensus.cc:2272] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.152202 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.151983 1179 ts_tablet_manager.cc:1916] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:14.163671 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9fe3e9a507441b8ba5953c0120289c3 consensus state has no leader
I20251029 08:52:14.164103 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_508e306ea26d403299fafa0a0cd2419c" } modify_external_catalogs: true
I20251029 08:52:14.164278 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_508e306ea26d403299fafa0a0cd2419c" } modify_external_catalogs: true
I20251029 08:52:14.165130 1183 tablet_bootstrap.cc:492] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.165356 1183 ts_tablet_manager.cc:1403] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20251029 08:52:14.166316 1183 raft_consensus.cc:359] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.166658 1175 ts_tablet_manager.cc:1916] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.166683 1183 raft_consensus.cc:385] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.166895 1183 raft_consensus.cc:740] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.167251 1183 consensus_queue.cc:260] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.170814 1177 ts_tablet_manager.cc:1434] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20251029 08:52:14.172477 1183 ts_tablet_manager.cc:1434] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:14.174119 1177 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.178972 1178 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.181715 1178 ts_tablet_manager.cc:1403] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.412s user 0.070s sys 0.112s
I20251029 08:52:14.182229 1183 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.183881 1183 tablet_bootstrap.cc:654] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.185467 1820 raft_consensus.cc:493] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.185592 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f" } modify_external_catalogs: true
I20251029 08:52:14.185604 1820 raft_consensus.cc:515] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.185732 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 39fc9612689b46d498d6092bd552da4e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.185814 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f" } modify_external_catalogs: true
I20251029 08:52:14.186240 1820 leader_election.cc:290] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.186297 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_23cbf2fd266c45efbdb3447e52d13f9a" } modify_external_catalogs: true
I20251029 08:52:14.186437 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_23cbf2fd266c45efbdb3447e52d13f9a" } modify_external_catalogs: true
W20251029 08:52:14.187744 861 leader_election.cc:343] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 94738d48801a4fb4a7ad1f5c4022afe1
I20251029 08:52:14.189226 1178 raft_consensus.cc:359] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.190779 1177 tablet_bootstrap.cc:654] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.187484 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94738d48801a4fb4a7ad1f5c4022afe1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.188268 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 55842c18dbef4513a61dd9629f2e5a4c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e2773cae9814df79ee60ea8b0fe4e9d) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:14.193755 861 leader_election.cc:343] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.193876 861 leader_election.cc:304] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.194521 1822 raft_consensus.cc:2749] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.199230 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:14.200708 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.201081 1178 raft_consensus.cc:385] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.201187 1179 ts_tablet_manager.cc:1929] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:14.200739 756 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.201349 1179 log.cc:1199] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/71ba5e1248dc4d4a8a4d7bb05b661a3f
I20251029 08:52:14.201782 1179 ts_tablet_manager.cc:1950] T 71ba5e1248dc4d4a8a4d7bb05b661a3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.202605 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 511) took 2363 ms. Trace:
I20251029 08:52:14.201220 1178 raft_consensus.cc:740] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.202708 1179 rpcz_store.cc:276] 1029 08:52:11.838783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.839343 (+ 560us) service_pool.cc:225] Handling call
1029 08:52:14.149863 (+2310520us) ts_tablet_manager.cc:1151] Deleting tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f
1029 08:52:14.149873 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.161326 (+ 11453us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.200259 (+ 38933us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.202552 (+ 2293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2304821}
I20251029 08:52:14.203154 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 71ba5e1248dc4d4a8a4d7bb05b661a3f (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:14.203635 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 513) took 2364 ms. Trace:
I20251029 08:52:14.203770 1179 rpcz_store.cc:276] 1029 08:52:11.838994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.839539 (+ 545us) service_pool.cc:225] Handling call
1029 08:52:14.203470 (+2363931us) ts_tablet_manager.cc:1151] Deleting tablet 633ace1413464e939e4e47782370e281
1029 08:52:14.203479 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.203590 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2358394}
W20251029 08:52:14.204280 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 633ace1413464e939e4e47782370e281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 633ace1413464e939e4e47782370e281 already in progress: creating tablet
I20251029 08:52:14.204602 1178 consensus_queue.cc:260] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.204720 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 509) took 2367 ms. Trace:
I20251029 08:52:14.204808 1179 rpcz_store.cc:276] 1029 08:52:11.838526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.839686 (+ 1160us) service_pool.cc:225] Handling call
1029 08:52:14.204578 (+2364892us) ts_tablet_manager.cc:1151] Deleting tablet f5b52314139945dd9c2e6ec899bf06dc
1029 08:52:14.204587 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.204673 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359450}
I20251029 08:52:14.205407 1178 ts_tablet_manager.cc:1434] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251029 08:52:14.208348 1178 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.208660 588 catalog_manager.cc:5649] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:14.209157 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]). Expected version 2 got 0
I20251029 08:52:14.201056 902 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.201370 1820 raft_consensus.cc:493] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.209651 1820 raft_consensus.cc:515] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:14.209169 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f5b52314139945dd9c2e6ec899bf06dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b52314139945dd9c2e6ec899bf06dc already in progress: creating tablet
I20251029 08:52:14.210284 1820 leader_election.cc:290] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.211407 1177 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.211403 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63b08840d0fa40428c61a05b277ad750" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.211525 1177 ts_tablet_manager.cc:1403] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
W20251029 08:52:14.211899 861 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:14.212128 861 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 63b08840d0fa40428c61a05b277ad750
I20251029 08:52:14.212219 861 leader_election.cc:304] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.212558 1820 raft_consensus.cc:2749] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:14.212952 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24834985a71349b9865840e9b5537d72 consensus state has no leader
I20251029 08:52:14.214548 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 514) took 2348 ms. Trace:
I20251029 08:52:14.214654 1179 rpcz_store.cc:276] 1029 08:52:11.864606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.864793 (+ 187us) service_pool.cc:225] Handling call
1029 08:52:14.212863 (+2348070us) ts_tablet_manager.cc:1151] Deleting tablet b1aaf233eba2461f9182e094a0115a4e
1029 08:52:14.212875 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.212965 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347848}
W20251029 08:52:14.215453 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b1aaf233eba2461f9182e094a0115a4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1aaf233eba2461f9182e094a0115a4e already in progress: creating tablet
I20251029 08:52:14.215787 1178 tablet_bootstrap.cc:654] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.220623 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ab04114f072046e2a09d1d5559571e7b" } new_table_name: "default.Table_c07c8c0bcb4b4c039d48ce3ea0ab9602" modify_external_catalogs: true
I20251029 08:52:14.221153 1179 tablet_replica.cc:333] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.221359 1179 raft_consensus.cc:2243] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.221549 1179 raft_consensus.cc:2272] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.222251 1177 raft_consensus.cc:359] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.222463 1177 raft_consensus.cc:385] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.222522 1177 raft_consensus.cc:740] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.222707 1177 consensus_queue.cc:260] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.225072 1179 ts_tablet_manager.cc:1916] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.225371 1177 ts_tablet_manager.cc:1434] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 08:52:14.228132 1177 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.228498 1774 raft_consensus.cc:493] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.228646 1774 raft_consensus.cc:515] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.229301 1774 leader_election.cc:290] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:14.229344 1779 raft_consensus.cc:515] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.229838 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.230367 720 leader_election.cc:343] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.231026 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:14.231732 718 leader_election.cc:343] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.231844 718 leader_election.cc:304] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:14.232070 1774 raft_consensus.cc:2749] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.234862 1183 tablet_bootstrap.cc:492] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.235085 1183 ts_tablet_manager.cc:1403] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.003s
I20251029 08:52:14.235894 1183 raft_consensus.cc:359] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.236164 1183 raft_consensus.cc:385] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.236276 1183 raft_consensus.cc:740] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.236593 1183 consensus_queue.cc:260] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.220692 1176 ts_tablet_manager.cc:1929] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.222344 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.238062 1176 log.cc:1199] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c9a1d5e1f68e4f0e8b78641bbdc98123
I20251029 08:52:14.240644 1779 leader_election.cc:290] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.241079 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9702a7d8b8304b49a250c129ecf28303" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.241145 1177 tablet_bootstrap.cc:654] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:14.241806 861 leader_election.cc:343] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:14.242676 1175 ts_tablet_manager.cc:1929] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.242776 1175 log.cc:1199] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:14.243054 1175 ts_tablet_manager.cc:1950] T 55842c18dbef4513a61dd9629f2e5a4c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.243804 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 581) took 1040 ms. Trace:
I20251029 08:52:14.243914 1175 rpcz_store.cc:276] 1029 08:52:13.203861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.205466 (+ 1605us) service_pool.cc:225] Handling call
1029 08:52:14.117739 (+912273us) ts_tablet_manager.cc:1151] Deleting tablet 55842c18dbef4513a61dd9629f2e5a4c
1029 08:52:14.117751 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.169985 (+ 52234us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.242554 (+ 72569us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.243786 (+ 1232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":912027}
W20251029 08:52:14.244443 861 leader_election.cc:343] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.244427 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 55842c18dbef4513a61dd9629f2e5a4c (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:14.244544 861 leader_election.cc:304] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.244920 1779 raft_consensus.cc:2749] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.245118 1176 ts_tablet_manager.cc:1950] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:14.246145 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
W20251029 08:52:14.246835 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9e1c023b679b4c3298bf953b53d4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1c023b679b4c3298bf953b53d4eb96 already in progress: creating tablet
W20251029 08:52:14.247419 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 107206cf33054e3fb880190eab8cd1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107206cf33054e3fb880190eab8cd1cc already in progress: creating tablet
W20251029 08:52:14.247992 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a506e35afe6a4c87901f27c434f3b348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a506e35afe6a4c87901f27c434f3b348 already in progress: creating tablet
W20251029 08:52:14.248562 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a8c4d57a59684723bfd27c7dda302ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4d57a59684723bfd27c7dda302ef3 already in progress: creating tablet
I20251029 08:52:14.249112 1177 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.250283 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 582) took 1037 ms. Trace:
I20251029 08:52:14.250388 1175 rpcz_store.cc:276] 1029 08:52:13.206700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.206754 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:14.244236 (+1037482us) ts_tablet_manager.cc:1151] Deleting tablet 7046bb1fcdd34d308aecddfdf0752911
1029 08:52:14.244245 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.244353 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037276}
I20251029 08:52:14.249346 1177 ts_tablet_manager.cc:1403] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 08:52:14.250903 1175 tablet_replica.cc:333] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.251080 1175 raft_consensus.cc:2243] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.251223 1175 raft_consensus.cc:2272] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.251813 1177 raft_consensus.cc:359] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:14.251968 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7046bb1fcdd34d308aecddfdf0752911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7046bb1fcdd34d308aecddfdf0752911 already in progress: creating tablet
I20251029 08:52:14.252041 1177 raft_consensus.cc:385] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.252187 1177 raft_consensus.cc:740] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:14.252261 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa0ef904c2fb4dfeadbf56ae659c4e20 already in progress: creating tablet
I20251029 08:52:14.252494 1177 consensus_queue.cc:260] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.255468 1175 ts_tablet_manager.cc:1916] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.256300 1177 ts_tablet_manager.cc:1434] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:14.256615 1177 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.258101 1183 ts_tablet_manager.cc:1434] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251029 08:52:14.261447 1183 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.261695 1175 ts_tablet_manager.cc:1929] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.261806 1175 log.cc:1199] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:14.261904 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6ec00cd64dc247da8442b6ba197f370a
I20251029 08:52:14.262138 1175 ts_tablet_manager.cc:1950] T 39dc5ebeb36043e0aa5955bacd06c2ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.262490 593 catalog_manager.cc:6719] ReplaceTablet: tablet 6ec00cd64dc247da8442b6ba197f370a deleted and replaced by tablet a4ae5978b71443a48a88bc69128b187d
I20251029 08:52:14.263149 1175 tablet_replica.cc:333] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.263265 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec00cd64dc247da8442b6ba197f370a with delete_type TABLET_DATA_DELETED (replaced by tablet a4ae5978b71443a48a88bc69128b187d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.263332 1175 raft_consensus.cc:2243] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:14.263387 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 39dc5ebeb36043e0aa5955bacd06c2ef (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
I20251029 08:52:14.263630 1175 raft_consensus.cc:2272] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.263664 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b8f1283f5a9245a599e5576a1e84b4a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f1283f5a9245a599e5576a1e84b4a5 already in progress: creating tablet
I20251029 08:52:14.263842 594 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6885ba12573a491e873b3089dc261139
I20251029 08:52:14.264317 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec00cd64dc247da8442b6ba197f370a with delete_type TABLET_DATA_DELETED (replaced by tablet a4ae5978b71443a48a88bc69128b187d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.264822 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec00cd64dc247da8442b6ba197f370a with delete_type TABLET_DATA_DELETED (replaced by tablet a4ae5978b71443a48a88bc69128b187d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.273808 1183 tablet_bootstrap.cc:654] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.281965 1177 tablet_bootstrap.cc:654] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.282133 1179 ts_tablet_manager.cc:1929] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.282275 1179 log.cc:1199] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a2947083e3e447d5addae5effd186eb8
I20251029 08:52:14.282665 1179 ts_tablet_manager.cc:1950] T a2947083e3e447d5addae5effd186eb8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:14.283278 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b8f2b34642704cd6bb966b6dcd494d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f2b34642704cd6bb966b6dcd494d51 already in progress: creating tablet
I20251029 08:52:14.283464 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 515) took 2340 ms. Trace:
I20251029 08:52:14.283574 1179 rpcz_store.cc:276] 1029 08:52:11.942684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.942845 (+ 161us) service_pool.cc:225] Handling call
1029 08:52:14.221045 (+2278200us) ts_tablet_manager.cc:1151] Deleting tablet a2947083e3e447d5addae5effd186eb8
1029 08:52:14.221054 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.271650 (+ 50596us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.281984 (+ 10334us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.283426 (+ 1442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277814}
I20251029 08:52:14.284008 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a2947083e3e447d5addae5effd186eb8 (table default.Table_affbc133198140a2b050f47a2d51c45d [id=4f4d2744cf5f444b8ab98d31b76e3988]) successfully deleted
W20251029 08:52:14.284355 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 07906300f1d54d4e8fde5165ee14c44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07906300f1d54d4e8fde5165ee14c44f already in progress: creating tablet
W20251029 08:52:14.284575 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ee8117e04f5f40e98ea755dc461067dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee8117e04f5f40e98ea755dc461067dd already in progress: creating tablet
W20251029 08:52:14.284842 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fe384ca428974ac5ba78e6506a1a8832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe384ca428974ac5ba78e6506a1a8832 already in progress: creating tablet
I20251029 08:52:14.284941 1176 tablet_replica.cc:333] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.285116 1176 raft_consensus.cc:2243] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.285274 1176 raft_consensus.cc:2272] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.287210 1779 raft_consensus.cc:493] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.287423 1779 raft_consensus.cc:515] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.288082 1779 leader_election.cc:290] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.288113 1178 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.288563 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b534aaa51b4fad9dc3fc915de947f1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.289047 861 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:14.289305 861 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:14.289409 861 leader_election.cc:304] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.289480 1178 ts_tablet_manager.cc:1403] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.001s
I20251029 08:52:14.290470 1178 raft_consensus.cc:359] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.290640 1178 raft_consensus.cc:385] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.290690 1178 raft_consensus.cc:740] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.291044 1779 raft_consensus.cc:2749] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.290977 1178 consensus_queue.cc:260] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.291476 1178 ts_tablet_manager.cc:1434] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:14.292081 1178 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.292178 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b11998f3ebf3402ab3f729d04fd5e0ef
I20251029 08:52:14.292853 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 9e1c023b679b4c3298bf953b53d4eb96 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.293356 1176 ts_tablet_manager.cc:1916] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.293690 902 tablet_service.cc:1552] Processing DeleteTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.294814 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.295825 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b49b69d8e23446a88c9e28445a8b4b56
I20251029 08:52:14.296514 902 tablet_service.cc:1552] Processing DeleteTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.297605 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.303722 1779 raft_consensus.cc:493] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.303869 1779 raft_consensus.cc:515] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.294178 756 tablet_service.cc:1552] Processing DeleteTablet for tablet b11998f3ebf3402ab3f729d04fd5e0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.297425 761 tablet_service.cc:1552] Processing DeleteTablet for tablet b49b69d8e23446a88c9e28445a8b4b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.304469 1779 leader_election.cc:290] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.303740 1175 ts_tablet_manager.cc:1916] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:14.305127 861 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:14.305508 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9165c4bf484a48b4a0b7246cb11571bf" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.306070 861 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.306183 861 leader_election.cc:304] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.306403 1779 raft_consensus.cc:2749] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.308434 1178 tablet_bootstrap.cc:654] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:14.309378 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3a62daad378430c9b593276685fa228 consensus state has no leader
I20251029 08:52:14.311159 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_81b1fe8277b3492cadd29290219544fc" } new_table_name: "default.Table_86364f11715846199b42645e6deeb2fb" modify_external_catalogs: true
I20251029 08:52:14.312721 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_98e2b0275a064384b92201c42be8a086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.313230 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98e2b0275a064384b92201c42be8a086 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.315737 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_31cc9d5473cd45dcb2a4fb9609ff132b" } new_table_name: "default.Table_6056be95de0a4b34ab9dcf62b2968444" modify_external_catalogs: true
I20251029 08:52:14.317143 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_40ff04cf3a70422293d9648f7364d8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.317683 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40ff04cf3a70422293d9648f7364d8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.319149 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_54ce98414ac0405db8a6149c47af342c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.319658 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54ce98414ac0405db8a6149c47af342c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.320057 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 517) took 2356 ms. Trace:
I20251029 08:52:14.320201 1179 rpcz_store.cc:276] 1029 08:52:11.963843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:11.963906 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:14.319854 (+2355948us) ts_tablet_manager.cc:1151] Deleting tablet eeabf8f23751447aa9af02363c40361d
1029 08:52:14.319863 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.320001 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2355408}
W20251029 08:52:14.320684 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet eeabf8f23751447aa9af02363c40361d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeabf8f23751447aa9af02363c40361d already in progress: creating tablet
I20251029 08:52:14.321681 1183 tablet_bootstrap.cc:492] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.321821 1183 ts_tablet_manager.cc:1403] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.002s
I20251029 08:52:14.322599 1183 raft_consensus.cc:359] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.322786 1183 raft_consensus.cc:385] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.322861 1183 raft_consensus.cc:740] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.323045 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2184d67d2edc408f939c2b4114a7eb15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.323091 1183 consensus_queue.cc:260] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:14.323493 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2184d67d2edc408f939c2b4114a7eb15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.324764 1178 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.324887 1178 ts_tablet_manager.cc:1403] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20251029 08:52:14.325578 1178 raft_consensus.cc:359] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.325739 1178 raft_consensus.cc:385] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.325790 1178 raft_consensus.cc:740] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.325965 1178 consensus_queue.cc:260] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.326243 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.326975 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c90eeb172173429785cab8c6674fbfcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.327443 1178 ts_tablet_manager.cc:1434] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251029 08:52:14.327450 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c90eeb172173429785cab8c6674fbfcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.327695 1178 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.329352 1178 tablet_bootstrap.cc:654] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.333124 1751 raft_consensus.cc:493] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.333336 1751 raft_consensus.cc:515] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.333976 1751 leader_election.cc:290] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:14.334262 1175 ts_tablet_manager.cc:1929] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.334388 1175 log.cc:1199] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/5f2dd9d6d0854e03a8298625938065c5
I20251029 08:52:14.334772 1175 ts_tablet_manager.cc:1950] T 5f2dd9d6d0854e03a8298625938065c5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.335078 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 586) took 1049 ms. Trace:
I20251029 08:52:14.335162 1175 rpcz_store.cc:276] 1029 08:52:13.285851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.285908 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:14.263114 (+977206us) ts_tablet_manager.cc:1151] Deleting tablet 5f2dd9d6d0854e03a8298625938065c5
1029 08:52:14.263120 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.308717 (+ 45597us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.334148 (+ 25431us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.335061 (+ 913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977019}
I20251029 08:52:14.335606 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 5f2dd9d6d0854e03a8298625938065c5 (table default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c]) successfully deleted
I20251029 08:52:14.336568 1175 tablet_replica.cc:333] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.336759 1175 raft_consensus.cc:2243] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.336898 1175 raft_consensus.cc:2272] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.337404 1175 ts_tablet_manager.cc:1916] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:14.338075 998 leader_election.cc:343] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7792ee223dbe43aa82990541caced045
W20251029 08:52:14.338465 996 leader_election.cc:343] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 7792ee223dbe43aa82990541caced045
I20251029 08:52:14.338614 996 leader_election.cc:304] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:14.339483 1751 raft_consensus.cc:2749] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:14.344836 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fb9ab76dcfe486e89bbe9391cab56f9 consensus state has no leader
I20251029 08:52:14.353574 1175 ts_tablet_manager.cc:1929] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.353705 1175 log.cc:1199] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:14.354030 1175 ts_tablet_manager.cc:1950] T 9702a7d8b8304b49a250c129ecf28303 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:14.354378 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62597189ecb047788b83788b514e9099 consensus state has no leader
I20251029 08:52:14.355161 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9702a7d8b8304b49a250c129ecf28303 (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
I20251029 08:52:14.355670 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_77649b9af0074613a28f9084f9f924d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.356179 759 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:14.356292 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77649b9af0074613a28f9084f9f924d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.359081 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.361042 1176 ts_tablet_manager.cc:1929] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.361065 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.361156 1176 log.cc:1199] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3295bb0b47714080a5c3f34049acd487
I20251029 08:52:14.361312 1183 ts_tablet_manager.cc:1434] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20251029 08:52:14.361932 1176 ts_tablet_manager.cc:1950] T 3295bb0b47714080a5c3f34049acd487 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:14.362960 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 3295bb0b47714080a5c3f34049acd487 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
W20251029 08:52:14.363663 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6ec00cd64dc247da8442b6ba197f370a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec00cd64dc247da8442b6ba197f370a already in progress: creating tablet
I20251029 08:52:14.364468 1176 tablet_replica.cc:333] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.365303 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 85ee033a724942b0a90f12f15cba2701 from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:14.367125 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9e1c023b679b4c3298bf953b53d4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1c023b679b4c3298bf953b53d4eb96 already in progress: creating tablet
I20251029 08:52:14.367092 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc9e5c781bc0463d9df56e2db965b1bb" } new_table_name: "default.Table_66d11c1d0bd449aeb31f85ed681c6051" modify_external_catalogs: true
I20251029 08:52:14.367427 1183 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.367647 1071 raft_consensus.cc:3060] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:14.367841 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 107206cf33054e3fb880190eab8cd1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107206cf33054e3fb880190eab8cd1cc already in progress: creating tablet
W20251029 08:52:14.368469 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a506e35afe6a4c87901f27c434f3b348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a506e35afe6a4c87901f27c434f3b348 already in progress: creating tablet
W20251029 08:52:14.369040 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a8c4d57a59684723bfd27c7dda302ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4d57a59684723bfd27c7dda302ef3 already in progress: creating tablet
I20251029 08:52:14.369539 1183 tablet_bootstrap.cc:654] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:14.369606 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b8f2b34642704cd6bb966b6dcd494d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f2b34642704cd6bb966b6dcd494d51 already in progress: creating tablet
I20251029 08:52:14.369900 1175 tablet_replica.cc:333] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.370143 1175 raft_consensus.cc:2243] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.370360 1175 raft_consensus.cc:2272] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.370836 1177 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.370898 1175 ts_tablet_manager.cc:1916] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.371065 1177 ts_tablet_manager.cc:1403] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.000s
I20251029 08:52:14.371805 1177 raft_consensus.cc:359] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.372062 1177 raft_consensus.cc:385] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.372031 1779 consensus_queue.cc:1048] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:14.372246 1176 raft_consensus.cc:2243] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:14.372536 1177 raft_consensus.cc:740] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.373015 1177 consensus_queue.cc:260] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.370090 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 520) took 2221 ms. Trace:
I20251029 08:52:14.375910 1177 ts_tablet_manager.cc:1434] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:14.376005 1179 rpcz_store.cc:276] 1029 08:52:12.148329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.148565 (+ 236us) service_pool.cc:225] Handling call
1029 08:52:14.369925 (+2221360us) ts_tablet_manager.cc:1151] Deleting tablet a2b7853b85d24a04b8b11caaa4fe8c4d
1029 08:52:14.369935 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.370042 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2220827}
I20251029 08:52:14.376493 1177 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:14.376515 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a2b7853b85d24a04b8b11caaa4fe8c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2b7853b85d24a04b8b11caaa4fe8c4d already in progress: creating tablet
I20251029 08:52:14.376991 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 521) took 2228 ms. Trace:
I20251029 08:52:14.377149 1179 rpcz_store.cc:276] 1029 08:52:12.148512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.148660 (+ 148us) service_pool.cc:225] Handling call
1029 08:52:14.376839 (+2228179us) ts_tablet_manager.cc:1151] Deleting tablet c7024e6eb4124109ad1575d216739fe2
1029 08:52:14.376850 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.376956 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2227539}
W20251029 08:52:14.377663 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c7024e6eb4124109ad1575d216739fe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7024e6eb4124109ad1575d216739fe2 already in progress: creating tablet
I20251029 08:52:14.378368 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 523) took 2171 ms. Trace:
I20251029 08:52:14.378577 1179 rpcz_store.cc:276] 1029 08:52:12.207026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.207251 (+ 225us) service_pool.cc:225] Handling call
1029 08:52:14.378162 (+2170911us) ts_tablet_manager.cc:1151] Deleting tablet 2ca466714c774fc9982ef145c6dd1571
1029 08:52:14.378174 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.378316 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170656}
W20251029 08:52:14.379058 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2ca466714c774fc9982ef145c6dd1571 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca466714c774fc9982ef145c6dd1571 already in progress: creating tablet
I20251029 08:52:14.379494 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 526) took 2160 ms. Trace:
I20251029 08:52:14.379642 1179 rpcz_store.cc:276] 1029 08:52:12.219284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.220111 (+ 827us) service_pool.cc:225] Handling call
1029 08:52:14.379334 (+2159223us) ts_tablet_manager.cc:1151] Deleting tablet 1dd62833c9d548adabbbf71856a4b181
1029 08:52:14.379343 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.379463 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121145}
W20251029 08:52:14.380136 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 1dd62833c9d548adabbbf71856a4b181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dd62833c9d548adabbbf71856a4b181 already in progress: creating tablet
I20251029 08:52:14.390273 1177 tablet_bootstrap.cc:654] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.391435 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3f6289022a94c0393bf7f32fb2439d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.392096 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3f6289022a94c0393bf7f32fb2439d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.393424 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0c27f6be2f8f414bb78ef1ffb99d8e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.393921 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c27f6be2f8f414bb78ef1ffb99d8e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.397331 1176 raft_consensus.cc:2272] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.398111 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29adc6d53af5419dbda1d41d75d37e71 consensus state has no leader
I20251029 08:52:14.399753 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
I20251029 08:52:14.401001 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.401141 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0f39ad7a803c4432ada5c9cf85ae0632" } modify_external_catalogs: true
I20251029 08:52:14.401374 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0f39ad7a803c4432ada5c9cf85ae0632" } modify_external_catalogs: true
I20251029 08:52:14.400452 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 530) took 1896 ms. Trace:
I20251029 08:52:14.402752 1179 rpcz_store.cc:276] 1029 08:52:12.503552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.505287 (+ 1735us) service_pool.cc:225] Handling call
1029 08:52:14.380630 (+1875343us) ts_tablet_manager.cc:1151] Deleting tablet e5fa181849474a559701fdb8c18d398b
1029 08:52:14.380642 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.400400 (+ 19758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1868444}
W20251029 08:52:14.403230 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e5fa181849474a559701fdb8c18d398b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5fa181849474a559701fdb8c18d398b already in progress: creating tablet
I20251029 08:52:14.403337 1176 ts_tablet_manager.cc:1916] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.404691 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 531) took 1893 ms. Trace:
I20251029 08:52:14.404829 1179 rpcz_store.cc:276] 1029 08:52:12.511383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.511973 (+ 590us) service_pool.cc:225] Handling call
1029 08:52:14.404488 (+1892515us) ts_tablet_manager.cc:1151] Deleting tablet 072cf27b0743488c8166e03b5e61c545
1029 08:52:14.404505 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.404644 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1887387}
W20251029 08:52:14.405246 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 072cf27b0743488c8166e03b5e61c545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072cf27b0743488c8166e03b5e61c545 already in progress: creating tablet
I20251029 08:52:14.406304 902 tablet_service.cc:1552] Processing DeleteTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb with delete_type TABLET_DATA_DELETED (replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.406787 1179 tablet_replica.cc:333] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.408246 1179 raft_consensus.cc:2243] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.408771 1179 raft_consensus.cc:2272] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.409631 1179 ts_tablet_manager.cc:1916] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.409641 1175 ts_tablet_manager.cc:1929] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.409749 1175 log.cc:1199] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:14.411134 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:14.411628 1175 ts_tablet_manager.cc:1950] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.412305 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
I20251029 08:52:14.412657 1175 tablet_replica.cc:333] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.412871 1175 raft_consensus.cc:2243] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.416522 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bc205fa7964840f9802b4bb2a9a4f954" } new_table_name: "default.Table_d4b6d93a144541b3b24624df6e5a0201" modify_external_catalogs: true
I20251029 08:52:14.416740 1175 raft_consensus.cc:2272] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.417354 1779 raft_consensus.cc:493] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.417474 1779 raft_consensus.cc:515] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.418066 1779 leader_election.cc:290] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:14.419737 902 tablet_service.cc:1505] Processing CreateTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 (DEFAULT_TABLE table=default.Table_892ad3428acd46749ff80438c5b8f0d3 [id=874a8e75e3e9463a81a1ffbb4c8e7889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.424190 1175 ts_tablet_manager.cc:1916] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.424378 899 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 (DEFAULT_TABLE table=default.Table_67e2e2e244b947f9b3b19e4c352c98e9 [id=f3dee45240ad47feb4cae2acb3d31022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.425510 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa348d6fa8c644bd8fe7b297a53c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.427064 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1af4bcd7af49a0b8a419298ab37ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.413671 1050 tablet_service.cc:1505] Processing CreateTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 (DEFAULT_TABLE table=default.Table_892ad3428acd46749ff80438c5b8f0d3 [id=874a8e75e3e9463a81a1ffbb4c8e7889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.430459 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa348d6fa8c644bd8fe7b297a53c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.432279 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:14.432488 781 raft_consensus.cc:2468] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:14.432763 1774 raft_consensus.cc:493] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.432874 1774 raft_consensus.cc:515] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.433416 1774 leader_election.cc:290] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:14.434212 761 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 (DEFAULT_TABLE table=default.Table_67e2e2e244b947f9b3b19e4c352c98e9 [id=f3dee45240ad47feb4cae2acb3d31022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.434571 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1af4bcd7af49a0b8a419298ab37ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.436619 861 leader_election.cc:304] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:14.436918 1822 raft_consensus.cc:2804] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:14.437005 1822 raft_consensus.cc:493] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:14.437065 1822 raft_consensus.cc:3060] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.437134 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc96ef1a94844a32923087c44a5bcfc8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.437649 720 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.438452 1179 ts_tablet_manager.cc:1929] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.438611 1179 log.cc:1199] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a9cf61d6137544019bca25acf2431986
I20251029 08:52:14.439090 1179 ts_tablet_manager.cc:1950] T a9cf61d6137544019bca25acf2431986 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.439682 1822 raft_consensus.cc:515] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.439888 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 534) took 1657 ms. Trace:
I20251029 08:52:14.440001 1179 rpcz_store.cc:276] 1029 08:52:12.782023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.782262 (+ 239us) service_pool.cc:225] Handling call
1029 08:52:14.406699 (+1624437us) ts_tablet_manager.cc:1151] Deleting tablet a9cf61d6137544019bca25acf2431986
1029 08:52:14.406712 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.434115 (+ 27403us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.438336 (+ 4221us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.439822 (+ 1486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1623240}
I20251029 08:52:14.440167 1820 raft_consensus.cc:493] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.440276 1820 raft_consensus.cc:515] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.440505 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a9cf61d6137544019bca25acf2431986 (table default.Table_61c7054042014cdb9fde5c728083e742 [id=f54c09d8818b4280b7cda7b4b4930cc0]) successfully deleted
I20251029 08:52:14.440810 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc96ef1a94844a32923087c44a5bcfc8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:14.440876 1820 leader_election.cc:290] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.440977 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 535) took 1573 ms. Trace:
I20251029 08:52:14.441072 1179 rpcz_store.cc:276] 1029 08:52:12.867702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.867759 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:14.440810 (+1573051us) ts_tablet_manager.cc:1151] Deleting tablet 57e2b19a227740a28ae601d77e62597b
1029 08:52:14.440820 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.440919 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1572796}
I20251029 08:52:14.441452 1822 leader_election.cc:290] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:14.441560 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28face87091b49d2971217618c95d2f5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.441602 718 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.441707 718 leader_election.cc:304] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:14.441751 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 57e2b19a227740a28ae601d77e62597b because the tablet was not found. No further retry: Not found: Tablet not found: 57e2b19a227740a28ae601d77e62597b
W20251029 08:52:14.442042 861 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.442142 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:14.442304 780 raft_consensus.cc:3060] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.442319 1179 tablet_replica.cc:333] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.442382 1774 raft_consensus.cc:2749] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.442562 1179 raft_consensus.cc:2243] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.442734 1179 raft_consensus.cc:2272] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.442955 759 tablet_service.cc:1505] Processing CreateTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 (DEFAULT_TABLE table=default.Table_892ad3428acd46749ff80438c5b8f0d3 [id=874a8e75e3e9463a81a1ffbb4c8e7889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.443253 1179 ts_tablet_manager.cc:1916] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.445120 780 raft_consensus.cc:2468] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:14.445194 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa348d6fa8c644bd8fe7b297a53c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.445564 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:14.446038 861 leader_election.cc:343] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:14.446238 861 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 28face87091b49d2971217618c95d2f5
I20251029 08:52:14.446341 861 leader_election.cc:304] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.446731 861 leader_election.cc:304] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:14.447185 1820 raft_consensus.cc:2749] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.447362 1820 raft_consensus.cc:2804] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:14.447458 1820 raft_consensus.cc:697] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:14.447707 1820 consensus_queue.cc:237] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.453217 1179 ts_tablet_manager.cc:1929] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.453390 1179 log.cc:1199] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a57b1dbbdf854bc6a6e1b5dc88b63bff
I20251029 08:52:14.453878 1179 ts_tablet_manager.cc:1950] T a57b1dbbdf854bc6a6e1b5dc88b63bff P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.454690 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 536) took 1578 ms. Trace:
I20251029 08:52:14.454816 1179 rpcz_store.cc:276] 1029 08:52:12.875978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.876171 (+ 193us) service_pool.cc:225] Handling call
1029 08:52:14.442224 (+1566053us) ts_tablet_manager.cc:1151] Deleting tablet a57b1dbbdf854bc6a6e1b5dc88b63bff
1029 08:52:14.442233 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.446764 (+ 4531us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.453086 (+ 6322us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.454654 (+ 1568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565813}
I20251029 08:52:14.455351 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a57b1dbbdf854bc6a6e1b5dc88b63bff (table default.Table_15261ce913ae430d8ea68e18a42df64a [id=a2015498bb844e87af722a69bcbf616a]) successfully deleted
I20251029 08:52:14.456068 1179 tablet_replica.cc:333] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.456317 1179 raft_consensus.cc:2243] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:14.456647 1179 pending_rounds.cc:70] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:14.456781 1179 pending_rounds.cc:77] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037613143949312 op_type: NO_OP noop_request { }
I20251029 08:52:14.456943 1179 raft_consensus.cc:2889] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:14.457098 1179 raft_consensus.cc:2272] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.457796 1179 ts_tablet_manager.cc:1916] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.459896 1183 tablet_bootstrap.cc:492] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.460160 1183 ts_tablet_manager.cc:1403] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.000s
I20251029 08:52:14.461061 1183 raft_consensus.cc:359] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.461355 1183 raft_consensus.cc:385] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.461480 1183 raft_consensus.cc:740] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.463862 1183 consensus_queue.cc:260] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.418748 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b07eb4befa41f3a1b5f2e9ebc2277c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.428345 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 (DEFAULT_TABLE table=default.Table_67e2e2e244b947f9b3b19e4c352c98e9 [id=f3dee45240ad47feb4cae2acb3d31022]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:14.465492 861 leader_election.cc:343] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.465858 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1af4bcd7af49a0b8a419298ab37ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.473168 1175 ts_tablet_manager.cc:1929] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.473335 1175 log.cc:1199] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/94738d48801a4fb4a7ad1f5c4022afe1
I20251029 08:52:14.473649 1175 ts_tablet_manager.cc:1950] T 94738d48801a4fb4a7ad1f5c4022afe1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.474234 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 94738d48801a4fb4a7ad1f5c4022afe1 (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:14.482486 1179 ts_tablet_manager.cc:1929] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:14.482905 1179 log.cc:1199] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/09759a50a82f4b758449da9fbbc1f72d
I20251029 08:52:14.483438 1179 ts_tablet_manager.cc:1950] T 09759a50a82f4b758449da9fbbc1f72d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.484344 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 537) took 1608 ms. Trace:
I20251029 08:52:14.484520 1179 rpcz_store.cc:276] 1029 08:52:12.876122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:12.876880 (+ 758us) service_pool.cc:225] Handling call
1029 08:52:14.455978 (+1579098us) ts_tablet_manager.cc:1151] Deleting tablet 09759a50a82f4b758449da9fbbc1f72d
1029 08:52:14.455988 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.463361 (+ 7373us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.482336 (+ 18975us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.484306 (+ 1970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1578860}
I20251029 08:52:14.485078 1175 tablet_replica.cc:333] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.485050 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 09759a50a82f4b758449da9fbbc1f72d (table default.Table_0f93256544fb4b56adbbcaf8c50ecb42 [id=dc9c30acd58449e4bcb9217b628b59dc]) successfully deleted
I20251029 08:52:14.485510 1175 raft_consensus.cc:2243] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.485729 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 558) took 1268 ms. Trace:
I20251029 08:52:14.485893 1175 raft_consensus.cc:2272] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.485855 1179 rpcz_store.cc:276] 1029 08:52:13.217629 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.228045 (+ 10416us) service_pool.cc:225] Handling call
1029 08:52:14.485570 (+1257525us) ts_tablet_manager.cc:1151] Deleting tablet 7046bb1fcdd34d308aecddfdf0752911
1029 08:52:14.485581 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.485690 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251627}
W20251029 08:52:14.486347 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7046bb1fcdd34d308aecddfdf0752911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7046bb1fcdd34d308aecddfdf0752911 already in progress: creating tablet
I20251029 08:52:14.486558 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 559) took 1071 ms. Trace:
I20251029 08:52:14.486752 596 master_service.cc:954] ReplaceTablet: received request to replace tablet ba201a54466240129d82a096f9316242 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.487054 1179 rpcz_store.cc:276] 1029 08:52:13.415089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.415188 (+ 99us) service_pool.cc:225] Handling call
1029 08:52:14.486380 (+1071192us) ts_tablet_manager.cc:1151] Deleting tablet aa0ef904c2fb4dfeadbf56ae659c4e20
1029 08:52:14.486391 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.486525 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038379}
W20251029 08:52:14.487739 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa0ef904c2fb4dfeadbf56ae659c4e20 already in progress: creating tablet
I20251029 08:52:14.488128 1179 tablet_replica.cc:333] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.488310 1175 ts_tablet_manager.cc:1916] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.488386 1179 raft_consensus.cc:2243] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.488611 1179 raft_consensus.cc:2272] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.489166 1179 ts_tablet_manager.cc:1916] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.491742 1779 consensus_queue.cc:1048] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:14.488384 1183 ts_tablet_manager.cc:1434] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251029 08:52:14.492758 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:14.493705 590 catalog_manager.cc:5649] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:14.493932 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a705dde62dcc4a949981ec95d2b75a98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.494218 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bc205fa7964840f9802b4bb2a9a4f954 [id=821160def5624241afdce0f734ef2411]). Expected version 2 got 0
W20251029 08:52:14.494598 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a705dde62dcc4a949981ec95d2b75a98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.494870 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.496438 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d22c6f8438dc46d79b03e79db3d79cac" } modify_external_catalogs: true
I20251029 08:52:14.496618 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d22c6f8438dc46d79b03e79db3d79cac" } modify_external_catalogs: true
I20251029 08:52:14.498581 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fc8d5092b4774b28adcf5f8a06dca48d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.499157 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc8d5092b4774b28adcf5f8a06dca48d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.498677 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79896e97f2934c50a068bf4beaf34966" } modify_external_catalogs: true
I20251029 08:52:14.500885 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79896e97f2934c50a068bf4beaf34966" } modify_external_catalogs: true
I20251029 08:52:14.501528 761 tablet_service.cc:1505] Processing CreateTablet for tablet 48853b956dd04e7c95d82f1d74da17a6 (DEFAULT_TABLE table=default.Table_5cfe4ea7c4e049d8af50f587af5f79f8 [id=6b69b40c9cc54f269d97d4196a59772d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.501965 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48853b956dd04e7c95d82f1d74da17a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.502789 1176 ts_tablet_manager.cc:1929] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.504339 899 tablet_service.cc:1505] Processing CreateTablet for tablet 48853b956dd04e7c95d82f1d74da17a6 (DEFAULT_TABLE table=default.Table_5cfe4ea7c4e049d8af50f587af5f79f8 [id=6b69b40c9cc54f269d97d4196a59772d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.504755 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48853b956dd04e7c95d82f1d74da17a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.507298 1175 ts_tablet_manager.cc:1929] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.507404 1175 log.cc:1199] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/28face87091b49d2971217618c95d2f5
I20251029 08:52:14.507684 1175 ts_tablet_manager.cc:1950] T 28face87091b49d2971217618c95d2f5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.508329 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 28face87091b49d2971217618c95d2f5 (table default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]) successfully deleted
I20251029 08:52:14.508630 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.508966 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.511034 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 48853b956dd04e7c95d82f1d74da17a6 (DEFAULT_TABLE table=default.Table_5cfe4ea7c4e049d8af50f587af5f79f8 [id=6b69b40c9cc54f269d97d4196a59772d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.511528 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48853b956dd04e7c95d82f1d74da17a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.511925 1183 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.517241 1179 ts_tablet_manager.cc:1929] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.517410 1179 log.cc:1199] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/5f2dd9d6d0854e03a8298625938065c5
I20251029 08:52:14.517880 1179 ts_tablet_manager.cc:1950] T 5f2dd9d6d0854e03a8298625938065c5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.518944 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 562) took 1099 ms. Trace:
I20251029 08:52:14.519044 1179 rpcz_store.cc:276] 1029 08:52:13.419105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.419622 (+ 517us) service_pool.cc:225] Handling call
1029 08:52:14.488059 (+1068437us) ts_tablet_manager.cc:1151] Deleting tablet 5f2dd9d6d0854e03a8298625938065c5
1029 08:52:14.488068 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.513846 (+ 25778us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.517132 (+ 3286us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.518895 (+ 1763us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":1037164}
I20251029 08:52:14.519527 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 5f2dd9d6d0854e03a8298625938065c5 (table default.Table_70d88123dfe248b6aeb0e6fa51fafc49 [id=6f5f02000b50491682d2ab710c4e1f5c]) successfully deleted
I20251029 08:52:14.520033 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 561) took 1102 ms. Trace:
I20251029 08:52:14.520224 1175 tablet_replica.cc:333] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.520130 1179 rpcz_store.cc:276] 1029 08:52:13.417527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.417627 (+ 100us) service_pool.cc:225] Handling call
1029 08:52:14.519850 (+1102223us) ts_tablet_manager.cc:1151] Deleting tablet b8f1283f5a9245a599e5576a1e84b4a5
1029 08:52:14.519860 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.519964 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068649}
I20251029 08:52:14.520475 1175 raft_consensus.cc:2243] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.520632 1175 raft_consensus.cc:2272] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.520725 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b8f1283f5a9245a599e5576a1e84b4a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f1283f5a9245a599e5576a1e84b4a5 already in progress: creating tablet
I20251029 08:52:14.521049 1179 tablet_replica.cc:333] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.521229 1179 raft_consensus.cc:2243] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.521284 1175 ts_tablet_manager.cc:1916] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.522461 1183 tablet_bootstrap.cc:654] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.533615 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4a756d6b5b3648e88cd2bd3400e51bed
I20251029 08:52:14.535187 1175 ts_tablet_manager.cc:1929] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.535429 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.502992 1178 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.536263 1178 ts_tablet_manager.cc:1403] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.003s
I20251029 08:52:14.505290 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.535982 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.537622 1178 raft_consensus.cc:359] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.537912 1178 raft_consensus.cc:385] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.538008 1178 raft_consensus.cc:740] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.538035 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a756d6b5b3648e88cd2bd3400e51bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.538722 1178 consensus_queue.cc:260] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.521401 1179 raft_consensus.cc:2272] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.539719 1179 ts_tablet_manager.cc:1916] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.541669 1178 ts_tablet_manager.cc:1434] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:14.542356 1178 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.546679 1178 tablet_bootstrap.cc:654] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.547909 1176 log.cc:1199] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b11998f3ebf3402ab3f729d04fd5e0ef
I20251029 08:52:14.535313 1175 log.cc:1199] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:14.548434 1175 ts_tablet_manager.cc:1950] T 9165c4bf484a48b4a0b7246cb11571bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.548482 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.549116 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9165c4bf484a48b4a0b7246cb11571bf (table default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]) successfully deleted
I20251029 08:52:14.548452 1176 ts_tablet_manager.cc:1950] T b11998f3ebf3402ab3f729d04fd5e0ef P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:14.549837 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 4deeec492a2545ca86583c6ea899a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.551147 1175 tablet_replica.cc:333] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.551342 1175 raft_consensus.cc:2243] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.551514 1175 raft_consensus.cc:2272] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.552057 1175 ts_tablet_manager.cc:1916] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.553992 1751 raft_consensus.cc:493] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.554165 1751 raft_consensus.cc:515] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.554386 1176 tablet_replica.cc:333] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.554651 1176 raft_consensus.cc:2243] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.554821 1751 leader_election.cc:290] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:14.554874 1176 raft_consensus.cc:2272] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.556263 996 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 4deeec492a2545ca86583c6ea899a0ab
W20251029 08:52:14.558099 998 leader_election.cc:343] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:14.558209 998 leader_election.cc:304] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:14.557750 1176 ts_tablet_manager.cc:1916] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.558640 1751 raft_consensus.cc:2749] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.559526 1183 tablet_bootstrap.cc:492] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.559728 1183 ts_tablet_manager.cc:1403] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.000s
I20251029 08:52:14.560708 1183 raft_consensus.cc:359] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.560940 1183 raft_consensus.cc:385] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.561004 1178 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.561115 1183 raft_consensus.cc:740] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.561206 1178 ts_tablet_manager.cc:1403] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251029 08:52:14.561472 1183 consensus_queue.cc:260] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.563591 1183 ts_tablet_manager.cc:1434] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:14.563926 1183 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.565063 1177 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.565192 1177 ts_tablet_manager.cc:1403] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.189s user 0.006s sys 0.000s
I20251029 08:52:14.565819 1177 raft_consensus.cc:359] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.565964 1177 raft_consensus.cc:385] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.566018 1177 raft_consensus.cc:740] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.566313 1177 consensus_queue.cc:260] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.568653 1179 ts_tablet_manager.cc:1929] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.569502 1177 ts_tablet_manager.cc:1434] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:14.569785 1177 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.570432 1179 log.cc:1199] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/64f73597d1764540bcaf472d3e890ee8
I20251029 08:52:14.570881 1179 ts_tablet_manager.cc:1950] T 64f73597d1764540bcaf472d3e890ee8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.573450 1178 raft_consensus.cc:359] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.573956 1175 ts_tablet_manager.cc:1929] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.574050 1175 log.cc:1199] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:14.574347 1175 ts_tablet_manager.cc:1950] T 23b534aaa51b4fad9dc3fc915de947f1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.574890 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 23b534aaa51b4fad9dc3fc915de947f1 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
I20251029 08:52:14.575555 1177 tablet_bootstrap.cc:654] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.576920 1178 raft_consensus.cc:385] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.577100 1178 raft_consensus.cc:740] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.577553 1178 consensus_queue.cc:260] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.581982 1178 ts_tablet_manager.cc:1434] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
W20251029 08:52:14.582832 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 07906300f1d54d4e8fde5165ee14c44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07906300f1d54d4e8fde5165ee14c44f already in progress: creating tablet
W20251029 08:52:14.583499 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ee8117e04f5f40e98ea755dc461067dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee8117e04f5f40e98ea755dc461067dd already in progress: creating tablet
I20251029 08:52:14.583760 1175 tablet_replica.cc:333] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.583940 1175 raft_consensus.cc:2243] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.583928 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.584123 1175 raft_consensus.cc:2272] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.585005 1175 ts_tablet_manager.cc:1916] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.586756 902 tablet_service.cc:1505] Processing CreateTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.587127 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f6ca7d4e534a46a1fba75acd7df39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.591163 1175 ts_tablet_manager.cc:1929] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.552760 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]) successfully deleted
I20251029 08:52:14.591305 1175 log.cc:1199] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/63b08840d0fa40428c61a05b277ad750
I20251029 08:52:14.591600 1175 ts_tablet_manager.cc:1950] T 63b08840d0fa40428c61a05b277ad750 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:14.592388 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b49b69d8e23446a88c9e28445a8b4b56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49b69d8e23446a88c9e28445a8b4b56 already in progress: creating tablet
I20251029 08:52:14.592769 1175 tablet_replica.cc:333] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.592955 1175 raft_consensus.cc:2243] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.593132 1175 raft_consensus.cc:2272] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.593148 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 63b08840d0fa40428c61a05b277ad750 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:14.593115 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec6a9c863b14029b8651e45aea5104e (DEFAULT_TABLE table=default.Table_081776ef8de44db6bec82f0f77a09aee [id=024f778e1dd443b19101681a442cbff9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:14.593410 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fe384ca428974ac5ba78e6506a1a8832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe384ca428974ac5ba78e6506a1a8832 already in progress: creating tablet
I20251029 08:52:14.593624 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec6a9c863b14029b8651e45aea5104e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.593613 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6ec00cd64dc247da8442b6ba197f370a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec00cd64dc247da8442b6ba197f370a already in progress: creating tablet
I20251029 08:52:14.593662 1175 ts_tablet_manager.cc:1916] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.594349 761 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec6a9c863b14029b8651e45aea5104e (DEFAULT_TABLE table=default.Table_081776ef8de44db6bec82f0f77a09aee [id=024f778e1dd443b19101681a442cbff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.594698 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec6a9c863b14029b8651e45aea5104e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.595840 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_081776ef8de44db6bec82f0f77a09aee" } modify_external_catalogs: true
I20251029 08:52:14.596033 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_081776ef8de44db6bec82f0f77a09aee" } modify_external_catalogs: true
I20251029 08:52:14.596298 1049 tablet_service.cc:1505] Processing CreateTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.596650 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f6ca7d4e534a46a1fba75acd7df39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.598822 759 tablet_service.cc:1505] Processing CreateTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.599115 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3cd2e9a95b6c40be9e17397a313d249b
I20251029 08:52:14.599160 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f6ca7d4e534a46a1fba75acd7df39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.606485 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (replaced by tablet b7ea4f1c81cd4ef6966ae0970dfb3fbb) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.608058 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (replaced by tablet b7ea4f1c81cd4ef6966ae0970dfb3fbb) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.609323 592 catalog_manager.cc:6719] ReplaceTablet: tablet 3cd2e9a95b6c40be9e17397a313d249b deleted and replaced by tablet b7ea4f1c81cd4ef6966ae0970dfb3fbb
I20251029 08:52:14.612418 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e687f593eead488db4bb445a63beeec7 (table default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff]). Expected version 1 got 0
I20251029 08:52:14.615162 899 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec6a9c863b14029b8651e45aea5104e (DEFAULT_TABLE table=default.Table_081776ef8de44db6bec82f0f77a09aee [id=024f778e1dd443b19101681a442cbff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.615557 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec6a9c863b14029b8651e45aea5104e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.570322 1183 tablet_bootstrap.cc:654] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.621475 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6f91eb858ae344549a78357e04f7e938 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.621821 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f91eb858ae344549a78357e04f7e938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.624054 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 560) took 1156 ms. Trace:
I20251029 08:52:14.624197 1179 rpcz_store.cc:276] 1029 08:52:13.415364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:13.415965 (+ 601us) service_pool.cc:225] Handling call
1029 08:52:14.520967 (+1105002us) ts_tablet_manager.cc:1151] Deleting tablet 64f73597d1764540bcaf472d3e890ee8
1029 08:52:14.520976 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:14.548143 (+ 27167us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.568522 (+ 20379us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:14.571655 (+ 3133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069413}
I20251029 08:52:14.626667 1179 tablet_replica.cc:333] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.626874 1179 raft_consensus.cc:2243] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.627027 1179 raft_consensus.cc:2272] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.627511 1177 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.627637 1177 ts_tablet_manager.cc:1403] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20251029 08:52:14.628314 1177 raft_consensus.cc:359] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.628468 1177 raft_consensus.cc:385] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.628526 1177 raft_consensus.cc:740] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.628710 1177 consensus_queue.cc:260] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.629529 1179 ts_tablet_manager.cc:1916] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.632560 1177 ts_tablet_manager.cc:1434] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:14.633847 1183 tablet_bootstrap.cc:492] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.634019 1183 ts_tablet_manager.cc:1403] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20251029 08:52:14.634352 1177 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.634786 1183 raft_consensus.cc:359] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.634991 1183 raft_consensus.cc:385] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.635071 1183 raft_consensus.cc:740] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.635353 1183 consensus_queue.cc:260] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.637380 1177 tablet_bootstrap.cc:654] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.639111 1183 ts_tablet_manager.cc:1434] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:14.639321 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 64f73597d1764540bcaf472d3e890ee8 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
W20251029 08:52:14.639670 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 107206cf33054e3fb880190eab8cd1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107206cf33054e3fb880190eab8cd1cc already in progress: creating tablet
W20251029 08:52:14.639978 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a506e35afe6a4c87901f27c434f3b348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a506e35afe6a4c87901f27c434f3b348 already in progress: creating tablet
W20251029 08:52:14.640264 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a8c4d57a59684723bfd27c7dda302ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4d57a59684723bfd27c7dda302ef3 already in progress: creating tablet
W20251029 08:52:14.640477 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9e1c023b679b4c3298bf953b53d4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1c023b679b4c3298bf953b53d4eb96 already in progress: creating tablet
I20251029 08:52:14.640705 1183 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.640705 1850 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:14.640815 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b8f2b34642704cd6bb966b6dcd494d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8f2b34642704cd6bb966b6dcd494d51 already in progress: creating tablet
W20251029 08:52:14.641005 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ee8117e04f5f40e98ea755dc461067dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee8117e04f5f40e98ea755dc461067dd already in progress: creating tablet
I20251029 08:52:14.641534 1851 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:14.642266 1848 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:14.645854 1183 tablet_bootstrap.cc:654] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.646139 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (replaced by tablet b7ea4f1c81cd4ef6966ae0970dfb3fbb) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.646889 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8392d4b37ead4dcaa819fe2c07fcb393 (DEFAULT_TABLE table=default.Table_745a04b358a243b4a7f0e0c7f3526da7 [id=f974c4ad9324463191c5f81e5f504368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.647258 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8392d4b37ead4dcaa819fe2c07fcb393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.650599 1179 ts_tablet_manager.cc:1929] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.650744 1179 log.cc:1199] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7792ee223dbe43aa82990541caced045
I20251029 08:52:14.651098 1179 ts_tablet_manager.cc:1950] T 7792ee223dbe43aa82990541caced045 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.652311 894 tablet_service.cc:1505] Processing CreateTablet for tablet ccf53a53fbf047329fd95f4deb752991 (DEFAULT_TABLE table=default.Table_9fdee9ea81294ee1806bada6c7a10d38 [id=0082514150964cbb9aa9234286c6d63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.652634 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf53a53fbf047329fd95f4deb752991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.653350 1179 tablet_replica.cc:333] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.653522 1179 raft_consensus.cc:2243] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.653689 1179 raft_consensus.cc:2272] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.654979 1179 ts_tablet_manager.cc:1916] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.658666 1178 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.659235 1183 tablet_bootstrap.cc:492] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.659385 1183 ts_tablet_manager.cc:1403] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20251029 08:52:14.659418 902 tablet_service.cc:1505] Processing CreateTablet for tablet 4d5b92342f7b49d290a9d2103114dd3b (DEFAULT_TABLE table=default.Table_090993e9452e40a68d67f5a3a00cf3e4 [id=dbbc6d00ffb546748575a0151eb74a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.660177 1183 raft_consensus.cc:359] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.660363 1183 raft_consensus.cc:385] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.660470 1183 raft_consensus.cc:740] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.660714 1183 consensus_queue.cc:260] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.662015 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5b92342f7b49d290a9d2103114dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.662817 1183 ts_tablet_manager.cc:1434] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:14.663076 1183 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.664772 1183 tablet_bootstrap.cc:654] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.667552 1175 ts_tablet_manager.cc:1929] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.667704 1175 log.cc:1199] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/b11998f3ebf3402ab3f729d04fd5e0ef
I20251029 08:52:14.668135 1175 ts_tablet_manager.cc:1950] T b11998f3ebf3402ab3f729d04fd5e0ef P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.669674 1175 tablet_replica.cc:333] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.671865 894 tablet_service.cc:1505] Processing CreateTablet for tablet a6d3b48876f04c48b4b2d0ac864ab7e5 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.672255 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d3b48876f04c48b4b2d0ac864ab7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.673645 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]) successfully deleted
W20251029 08:52:14.673918 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b49b69d8e23446a88c9e28445a8b4b56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49b69d8e23446a88c9e28445a8b4b56 already in progress: creating tablet
I20251029 08:52:14.674255 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 85ee033a724942b0a90f12f15cba2701
I20251029 08:52:14.674688 588 catalog_manager.cc:6719] ReplaceTablet: tablet 85ee033a724942b0a90f12f15cba2701 deleted and replaced by tablet a013257089774ba8b1f535e5f94ac1bb
I20251029 08:52:14.675009 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fd8c4e0f158b4582923e9de97aa89bcf" } new_table_name: "default.Table_f95f49b0100b4f0f9fdb61e2ff742ac9" modify_external_catalogs: true
I20251029 08:52:14.677224 896 tablet_service.cc:1505] Processing CreateTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 (DEFAULT_TABLE table=default.Table_b864bd5ff3e341938015d4bae27df6d4 [id=753bb0295ded4153b042b96f2d2e3f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.677635 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dcdd90e4a54cbfb0cb62192c465c40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.678725 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 07906300f1d54d4e8fde5165ee14c44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07906300f1d54d4e8fde5165ee14c44f already in progress: creating tablet
I20251029 08:52:14.679337 761 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.679917 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.681895 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2d05ded03f9d4415ada974d91d47ec0c" } new_table_name: "default.Table_57929518185c4558a0f9f12c40945be2" modify_external_catalogs: true
I20251029 08:52:14.684944 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ee033a724942b0a90f12f15cba2701 with delete_type TABLET_DATA_DELETED (replaced by tablet a013257089774ba8b1f535e5f94ac1bb) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.686136 756 tablet_service.cc:1505] Processing CreateTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 (DEFAULT_TABLE table=default.Table_b864bd5ff3e341938015d4bae27df6d4 [id=753bb0295ded4153b042b96f2d2e3f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.686492 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dcdd90e4a54cbfb0cb62192c465c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.686389 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1c4199c5e69245c5a9e8cae1a4ed98da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:14.679996 1175 raft_consensus.cc:2243] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.687186 1175 raft_consensus.cc:2272] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:14.688282 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c4199c5e69245c5a9e8cae1a4ed98da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.689690 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ba201a54466240129d82a096f9316242
I20251029 08:52:14.690126 596 catalog_manager.cc:6719] ReplaceTablet: tablet ba201a54466240129d82a096f9316242 deleted and replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f
I20251029 08:52:14.691934 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dd8cd3b63a734120b7e5d8da5bcad3e2" } new_table_name: "default.Table_bb70d01442d841199629d07f2ee8ed72" modify_external_catalogs: true
I20251029 08:52:14.692616 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:14.694123 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.694640 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8392d4b37ead4dcaa819fe2c07fcb393 (DEFAULT_TABLE table=default.Table_745a04b358a243b4a7f0e0c7f3526da7 [id=f974c4ad9324463191c5f81e5f504368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.695019 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8392d4b37ead4dcaa819fe2c07fcb393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.695240 591 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b7ea4f1c81cd4ef6966ae0970dfb3fbb
I20251029 08:52:14.695643 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5ec6a9c863b14029b8651e45aea5104e
I20251029 08:52:14.696270 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 24ab9df6f768410eb4f66cffdcd68df2 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.696516 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ec6a9c863b14029b8651e45aea5104e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.697695 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2f924d8dc47a422b99f57fbaea177d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.699028 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c18e0e01fdc54b8dab97ec339d2aaa07 consensus state has no leader
W20251029 08:52:14.699777 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f924d8dc47a422b99f57fbaea177d74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.700167 1820 raft_consensus.cc:493] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.700300 1820 raft_consensus.cc:515] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.700961 1820 leader_election.cc:290] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.701717 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_37caed2a62614c98a11dab8ae8b51b2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.702347 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37caed2a62614c98a11dab8ae8b51b2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.703848 780 raft_consensus.cc:1275] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:14.703862 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f4b5630f70e44d8886ee111768d79161"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.704344 861 leader_election.cc:343] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: be706771a95e45e5a20a15614eadfe1a
W20251029 08:52:14.704478 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4b5630f70e44d8886ee111768d79161 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.704623 1820 consensus_queue.cc:1048] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:14.705230 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e733598233c746a3aeab693a18a55a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.705798 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e733598233c746a3aeab693a18a55a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.705976 762 tablet_service.cc:1505] Processing CreateTablet for tablet ccf53a53fbf047329fd95f4deb752991 (DEFAULT_TABLE table=default.Table_9fdee9ea81294ee1806bada6c7a10d38 [id=0082514150964cbb9aa9234286c6d63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.706349 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf53a53fbf047329fd95f4deb752991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.707378 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e66832af5e114b158d59c2326322b87c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.707947 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e66832af5e114b158d59c2326322b87c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.709515 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cc6f14c92a63433f90c93e8bd5956878" } new_table_name: "default.Table_8b8706e0267f49478503d8a45662ac0b" modify_external_catalogs: true
I20251029 08:52:14.714319 894 tablet_service.cc:1505] Processing CreateTablet for tablet 36bea657aa9044f88e50bf4e749ff01e (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.714833 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bea657aa9044f88e50bf4e749ff01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.715415 1175 ts_tablet_manager.cc:1916] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.716216 1774 raft_consensus.cc:493] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.716341 1774 raft_consensus.cc:515] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.716912 1774 leader_election.cc:290] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:14.717574 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:14.718282 718 leader_election.cc:343] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.722607 759 tablet_service.cc:1505] Processing CreateTablet for tablet 6f91eb858ae344549a78357e04f7e938 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.723037 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f91eb858ae344549a78357e04f7e938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.726802 895 tablet_service.cc:1505] Processing CreateTablet for tablet dbfcb4e264cb401ab88ed9437775ad98 (DEFAULT_TABLE table=default.Table_87f770c9be434c09a9ddef0f8768c953 [id=33162e044e3c46ba81a4957ded99fad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.727037 761 tablet_service.cc:1505] Processing CreateTablet for tablet 4d5b92342f7b49d290a9d2103114dd3b (DEFAULT_TABLE table=default.Table_090993e9452e40a68d67f5a3a00cf3e4 [id=dbbc6d00ffb546748575a0151eb74a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.727242 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb4e264cb401ab88ed9437775ad98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.727371 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5b92342f7b49d290a9d2103114dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.729686 1178 tablet_bootstrap.cc:654] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.731194 888 tablet_service.cc:1505] Processing CreateTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 (DEFAULT_TABLE table=default.Table_0a016861cf8d4f54a567239750fc81df [id=58f69b1a32464bb2b1ed828c7c233452]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.731611 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd357a723a1d4bc1aa4b480c8a29ef03. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.740170 861 consensus_peers.cc:597] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:14.740895 889 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.741684 756 tablet_service.cc:1505] Processing CreateTablet for tablet a6d3b48876f04c48b4b2d0ac864ab7e5 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.742053 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d3b48876f04c48b4b2d0ac864ab7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.746023 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be706771a95e45e5a20a15614eadfe1a" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.746445 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.746870 1177 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
W20251029 08:52:14.746825 861 leader_election.cc:343] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.746928 861 leader_election.cc:304] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.746976 1177 ts_tablet_manager.cc:1403] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.113s user 0.002s sys 0.000s
I20251029 08:52:14.747272 1820 raft_consensus.cc:2749] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.747745 1177 raft_consensus.cc:359] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.747897 1177 raft_consensus.cc:385] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.747957 1177 raft_consensus.cc:740] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.749118 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 8392d4b37ead4dcaa819fe2c07fcb393 (DEFAULT_TABLE table=default.Table_745a04b358a243b4a7f0e0c7f3526da7 [id=f974c4ad9324463191c5f81e5f504368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.749615 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8392d4b37ead4dcaa819fe2c07fcb393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.750505 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 6f91eb858ae344549a78357e04f7e938 (DEFAULT_TABLE table=default.Table_d782dc9366fe4e2bbc42fe62f589e7b5 [id=acab1181d22d454abea6817e3481f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.750934 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f91eb858ae344549a78357e04f7e938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.751370 1177 consensus_queue.cc:260] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.751976 1177 ts_tablet_manager.cc:1434] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:14.752830 1033 tablet_service.cc:1505] Processing CreateTablet for tablet ccf53a53fbf047329fd95f4deb752991 (DEFAULT_TABLE table=default.Table_9fdee9ea81294ee1806bada6c7a10d38 [id=0082514150964cbb9aa9234286c6d63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.754434 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf53a53fbf047329fd95f4deb752991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.757216 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 (DEFAULT_TABLE table=default.Table_b864bd5ff3e341938015d4bae27df6d4 [id=753bb0295ded4153b042b96f2d2e3f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.757463 1175 ts_tablet_manager.cc:1929] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.757579 1175 log.cc:1199] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:14.757869 1175 ts_tablet_manager.cc:1950] T 7cbce5d595364b4bad36d59fb532c237 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:14.757921 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76dcdd90e4a54cbfb0cb62192c465c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.747610 1176 ts_tablet_manager.cc:1929] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:14.748442 720 leader_election.cc:343] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.758205 720 leader_election.cc:304] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:14.758653 1774 raft_consensus.cc:2749] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.760108 1176 log.cc:1199] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:14.760430 1176 ts_tablet_manager.cc:1950] T bc96ef1a94844a32923087c44a5bcfc8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:14.761811 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ee033a724942b0a90f12f15cba2701 with delete_type TABLET_DATA_DELETED (replaced by tablet a013257089774ba8b1f535e5f94ac1bb) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:14.762533 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58932f38c0d64aec9a02238eaea56a25 consensus state has no leader
I20251029 08:52:14.763365 1179 ts_tablet_manager.cc:1929] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:14.763495 1179 log.cc:1199] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/55842c18dbef4513a61dd9629f2e5a4c
I20251029 08:52:14.763875 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ee033a724942b0a90f12f15cba2701 with delete_type TABLET_DATA_DELETED (replaced by tablet a013257089774ba8b1f535e5f94ac1bb) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.764106 1179 ts_tablet_manager.cc:1950] T 55842c18dbef4513a61dd9629f2e5a4c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.765285 1177 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.765708 1179 tablet_replica.cc:333] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.765954 1179 raft_consensus.cc:2243] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.766182 1179 raft_consensus.cc:2272] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.766804 1179 ts_tablet_manager.cc:1916] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.769731 1177 tablet_bootstrap.cc:654] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.770102 888 tablet_service.cc:1505] Processing CreateTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 (DEFAULT_TABLE table=default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.770430 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92cc4a93a584dffaafe21316fe8dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.773485 1820 raft_consensus.cc:493] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.773612 1820 raft_consensus.cc:515] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.774180 1820 leader_election.cc:290] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.774820 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:14.775005 781 raft_consensus.cc:2468] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:14.775310 861 leader_election.cc:304] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:14.775560 1820 raft_consensus.cc:2804] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:14.775619 1820 raft_consensus.cc:493] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:14.775671 1820 raft_consensus.cc:3060] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.778343 1820 raft_consensus.cc:515] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.778880 1820 leader_election.cc:290] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.779556 1751 raft_consensus.cc:493] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.779542 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:14.779513 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc19ba1ac054fa3b459aa62a26c1024 (DEFAULT_TABLE table=default.Table_2fa23a8b35304536899bbbc02a070691 [id=aeb63b19792f4efe86b5244c6509a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.779747 781 raft_consensus.cc:3060] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.779910 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:14.780000 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc19ba1ac054fa3b459aa62a26c1024. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.780349 861 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.782431 1820 raft_consensus.cc:493] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.782557 1820 raft_consensus.cc:515] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.783038 781 raft_consensus.cc:2468] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:14.783111 1820 leader_election.cc:290] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.783249 1177 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.783375 1177 ts_tablet_manager.cc:1403] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251029 08:52:14.783443 1751 raft_consensus.cc:515] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:14.783564 861 leader_election.cc:304] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:14.783715 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:14.783856 781 raft_consensus.cc:2468] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:14.783882 1820 raft_consensus.cc:2804] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:14.784003 1820 raft_consensus.cc:697] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:14.784086 1177 raft_consensus.cc:359] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.784258 1177 raft_consensus.cc:385] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.784363 1177 raft_consensus.cc:740] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.784269 1820 consensus_queue.cc:237] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.784710 1177 consensus_queue.cc:260] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.785209 1177 ts_tablet_manager.cc:1434] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:14.784371 861 leader_election.cc:304] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:14.785537 1177 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.785598 1822 raft_consensus.cc:2804] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:14.785705 1822 raft_consensus.cc:493] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:14.785823 1822 raft_consensus.cc:3060] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:14.788977 1177 tablet_bootstrap.cc:654] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.789999 1822 raft_consensus.cc:515] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.790654 1822 leader_election.cc:290] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.790915 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.791136 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:14.791307 781 raft_consensus.cc:3060] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:14.791877 998 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:14.792291 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:14.792855 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:14.793201 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 4d5b92342f7b49d290a9d2103114dd3b (DEFAULT_TABLE table=default.Table_090993e9452e40a68d67f5a3a00cf3e4 [id=dbbc6d00ffb546748575a0151eb74a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.793655 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5b92342f7b49d290a9d2103114dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.794133 781 raft_consensus.cc:2468] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:14.795701 861 leader_election.cc:304] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:14.796032 861 leader_election.cc:343] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.796140 1822 raft_consensus.cc:2804] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:14.796347 861 leader_election.cc:343] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.796392 1822 raft_consensus.cc:697] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
W20251029 08:52:14.796619 861 leader_election.cc:343] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.796682 1822 consensus_queue.cc:237] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.798206 1751 leader_election.cc:290] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:14.801146 1177 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:14.801311 1177 ts_tablet_manager.cc:1403] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20251029 08:52:14.802086 1177 raft_consensus.cc:359] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.802253 1177 raft_consensus.cc:385] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.802358 1177 raft_consensus.cc:740] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:14.802599 996 leader_election.cc:343] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:14.802567 1177 consensus_queue.cc:260] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.802711 996 leader_election.cc:304] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:14.803202 1177 ts_tablet_manager.cc:1434] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:14.803261 1751 raft_consensus.cc:2749] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.803499 1177 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:14.805161 1177 tablet_bootstrap.cc:654] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.806618 590 master_service.cc:954] ReplaceTablet: received request to replace tablet add0605edf5c4511ae4c8d9cb7f0babb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:14.807020 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_64a621b6339340e99357f629eeb43501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.807664 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64a621b6339340e99357f629eeb43501 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.809118 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7792ee223dbe43aa82990541caced045 (table default.Table_9b80aae43bb845d1bb98687e0e6116f6 [id=335946e15b7c424ba3b769c46bd76ee9]) successfully deleted
W20251029 08:52:14.809386 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 45609.7us
I20251029 08:52:14.812839 759 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.813777 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ec6a9c863b14029b8651e45aea5104e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.814146 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:14.814739 759 tablet_service.cc:1505] Processing CreateTablet for tablet 36bea657aa9044f88e50bf4e749ff01e (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.815202 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bea657aa9044f88e50bf4e749ff01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.816670 756 tablet_service.cc:1505] Processing CreateTablet for tablet dbfcb4e264cb401ab88ed9437775ad98 (DEFAULT_TABLE table=default.Table_87f770c9be434c09a9ddef0f8768c953 [id=33162e044e3c46ba81a4957ded99fad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.816999 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb4e264cb401ab88ed9437775ad98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.817026 763 tablet_service.cc:1505] Processing CreateTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 (DEFAULT_TABLE table=default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.817437 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92cc4a93a584dffaafe21316fe8dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.822480 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5ba2c461229a40f190ba08563f21c14f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:14.823083 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ba2c461229a40f190ba08563f21c14f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:14.830829 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.831228 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ec6a9c863b14029b8651e45aea5104e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.831914 1051 tablet_service.cc:1505] Processing CreateTablet for tablet a6d3b48876f04c48b4b2d0ac864ab7e5 (DEFAULT_TABLE table=default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.832432 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d3b48876f04c48b4b2d0ac864ab7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.836925 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:14.837196 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 36bea657aa9044f88e50bf4e749ff01e (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.837394 589 catalog_manager.cc:6719] ReplaceTablet: tablet 24ab9df6f768410eb4f66cffdcd68df2 deleted and replaced by tablet 5e35d40b730a4a7883c324a6cc93b161
I20251029 08:52:14.837563 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bea657aa9044f88e50bf4e749ff01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.838376 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:14.840632 1037 tablet_service.cc:1505] Processing CreateTablet for tablet dbfcb4e264cb401ab88ed9437775ad98 (DEFAULT_TABLE table=default.Table_87f770c9be434c09a9ddef0f8768c953 [id=33162e044e3c46ba81a4957ded99fad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.841042 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb4e264cb401ab88ed9437775ad98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.844933 1048 tablet_service.cc:1505] Processing CreateTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 (DEFAULT_TABLE table=default.Table_0a016861cf8d4f54a567239750fc81df [id=58f69b1a32464bb2b1ed828c7c233452]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.845432 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd357a723a1d4bc1aa4b480c8a29ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.848769 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc19ba1ac054fa3b459aa62a26c1024 (DEFAULT_TABLE table=default.Table_2fa23a8b35304536899bbbc02a070691 [id=aeb63b19792f4efe86b5244c6509a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.849071 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc19ba1ac054fa3b459aa62a26c1024. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.849229 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0073b1df0b7d4647bff7d80fe246edf6" } modify_external_catalogs: true
I20251029 08:52:14.849500 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0073b1df0b7d4647bff7d80fe246edf6" } modify_external_catalogs: true
I20251029 08:52:14.852638 1039 tablet_service.cc:1505] Processing CreateTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 (DEFAULT_TABLE table=default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.852948 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92cc4a93a584dffaafe21316fe8dc06. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:14.858618 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb80603843ab428d8329b5c387a28a7b has no consensus state
I20251029 08:52:14.873667 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_06f02760c1014a1dbe9f6eea969646a9" } modify_external_catalogs: true
I20251029 08:52:14.873905 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_06f02760c1014a1dbe9f6eea969646a9" } modify_external_catalogs: true
I20251029 08:52:14.874730 781 raft_consensus.cc:1275] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 08:52:14.875062 861 consensus_peers.cc:597] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:14.875357 1779 consensus_queue.cc:1048] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:14.879802 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 55842c18dbef4513a61dd9629f2e5a4c (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
W20251029 08:52:14.880079 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fe384ca428974ac5ba78e6506a1a8832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe384ca428974ac5ba78e6506a1a8832 already in progress: creating tablet
I20251029 08:52:14.891013 751 tablet_service.cc:1505] Processing CreateTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 (DEFAULT_TABLE table=default.Table_0a016861cf8d4f54a567239750fc81df [id=58f69b1a32464bb2b1ed828c7c233452]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.892213 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.896137 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd357a723a1d4bc1aa4b480c8a29ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.896454 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.904134 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu pid 1262
I20251029 08:52:14.894519 762 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc19ba1ac054fa3b459aa62a26c1024 (DEFAULT_TABLE table=default.Table_2fa23a8b35304536899bbbc02a070691 [id=aeb63b19792f4efe86b5244c6509a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:14.906526 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc19ba1ac054fa3b459aa62a26c1024. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:14.932075 1779 raft_consensus.cc:493] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.932267 1779 raft_consensus.cc:515] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.932971 1779 leader_election.cc:290] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.933385 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8daa66608cf94913afb94ef661b3c898" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.933776 861 leader_election.cc:343] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 8daa66608cf94913afb94ef661b3c898
W20251029 08:52:14.934037 861 leader_election.cc:343] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.934142 861 leader_election.cc:304] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:14.934374 1779 raft_consensus.cc:2749] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:14.946691 1178 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:14.946859 1178 ts_tablet_manager.cc:1403] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.288s user 0.001s sys 0.005s
I20251029 08:52:14.947731 1178 raft_consensus.cc:359] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.947912 1178 raft_consensus.cc:385] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.947968 1178 raft_consensus.cc:740] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.948199 1183 tablet_bootstrap.cc:492] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:14.948184 1178 consensus_queue.cc:260] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.948379 1183 ts_tablet_manager.cc:1403] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.285s user 0.020s sys 0.016s
I20251029 08:52:14.948611 1178 ts_tablet_manager.cc:1434] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:14.948884 1178 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:14.949254 1183 raft_consensus.cc:359] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.949494 1183 raft_consensus.cc:385] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:14.949586 1183 raft_consensus.cc:740] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:14.950035 1183 consensus_queue.cc:260] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:14.952762 1183 ts_tablet_manager.cc:1434] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:14.953076 1183 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:14.956972 1178 tablet_bootstrap.cc:654] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:14.964114 1863 raft_consensus.cc:493] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:14.964291 1863 raft_consensus.cc:515] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:14.965304 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 58932f38c0d64aec9a02238eaea56a25 (table default.Table_57929518185c4558a0f9f12c40945be2 [id=60ed38dd0d6e4bc5851de7c04b92abfc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 58932f38c0d64aec9a02238eaea56a25 consensus state has no leader
I20251029 08:52:14.965253 1848 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:14.965363 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:14.968304 861 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:14.970186 1848 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:14.976783 1863 leader_election.cc:290] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:14.976863 1176 tablet_replica.cc:333] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.977049 1176 raft_consensus.cc:2243] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.977272 1176 raft_consensus.cc:2272] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.977809 1176 ts_tablet_manager.cc:1916] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:14.880317 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6ec00cd64dc247da8442b6ba197f370a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec00cd64dc247da8442b6ba197f370a already in progress: creating tablet
W20251029 08:52:14.978966 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 98663.1us
I20251029 08:52:14.980144 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:14.980345 784 raft_consensus.cc:2468] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:14.981323 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:14.985251 1176 ts_tablet_manager.cc:1929] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:14.985359 1176 log.cc:1199] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4a756d6b5b3648e88cd2bd3400e51bed
I20251029 08:52:14.985342 1179 ts_tablet_manager.cc:1929] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:14.985472 1179 log.cc:1199] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b11998f3ebf3402ab3f729d04fd5e0ef
I20251029 08:52:14.985718 1176 ts_tablet_manager.cc:1950] T 4a756d6b5b3648e88cd2bd3400e51bed P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:14.985860 1179 ts_tablet_manager.cc:1950] T b11998f3ebf3402ab3f729d04fd5e0ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:14.987931 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_d4b6d93a144541b3b24624df6e5a0201 [id=821160def5624241afdce0f734ef2411]). Expected version 3 got 0
I20251029 08:52:14.988040 1183 tablet_bootstrap.cc:654] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:14.988282 1850 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.347s user 0.001s sys 0.000s
I20251029 08:52:14.989707 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7cbce5d595364b4bad36d59fb532c237 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
I20251029 08:52:14.990053 1175 tablet_replica.cc:333] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:14.992996 1176 tablet_replica.cc:333] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:14.993513 1176 raft_consensus.cc:2243] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.994868 1176 raft_consensus.cc:2272] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.995061 1179 tablet_replica.cc:333] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:14.995313 1179 raft_consensus.cc:2243] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:14.995527 1179 raft_consensus.cc:2272] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:14.996209 1179 ts_tablet_manager.cc:1916] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:14.996904 1176 ts_tablet_manager.cc:1916] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.005728 1871 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:15.015296 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.016827 861 leader_election.cc:304] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:15.017133 1177 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.017307 1183 tablet_bootstrap.cc:492] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.017520 1183 ts_tablet_manager.cc:1403] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20251029 08:52:15.018424 1177 ts_tablet_manager.cc:1403] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.214s user 0.029s sys 0.025s
I20251029 08:52:15.018471 1183 raft_consensus.cc:359] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.018865 1183 raft_consensus.cc:385] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.019003 1183 raft_consensus.cc:740] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.019340 1175 raft_consensus.cc:2243] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.019459 1183 consensus_queue.cc:260] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.019999 1868 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:15.020797 1177 raft_consensus.cc:359] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.021806 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bc96ef1a94844a32923087c44a5bcfc8 (table default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]) successfully deleted
I20251029 08:52:15.022217 1179 ts_tablet_manager.cc:1929] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.022357 1179 log.cc:1199] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2e96cbdc4b7845b3823617a8e9a8c8b2
I20251029 08:52:15.024993 1183 ts_tablet_manager.cc:1434] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:15.028133 1177 raft_consensus.cc:385] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.028254 1177 raft_consensus.cc:740] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.028528 1863 raft_consensus.cc:2804] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:15.030288 1183 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.032917 1179 ts_tablet_manager.cc:1950] T 2e96cbdc4b7845b3823617a8e9a8c8b2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.033658 1183 tablet_bootstrap.cc:654] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.036278 1175 raft_consensus.cc:2272] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.036339 1179 tablet_replica.cc:333] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.036584 1179 raft_consensus.cc:2243] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.036793 1179 raft_consensus.cc:2272] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.036967 1177 consensus_queue.cc:260] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.037317 1179 ts_tablet_manager.cc:1916] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.037518 1177 ts_tablet_manager.cc:1434] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251029 08:52:15.038412 1177 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.032879 1774 raft_consensus.cc:493] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.039717 1774 raft_consensus.cc:515] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.040338 1774 leader_election.cc:290] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.041013 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215cd666c71a4179a19e298868a88e96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.041478 720 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.045189 1863 raft_consensus.cc:493] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:15.045325 1863 raft_consensus.cc:3060] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:15.045384 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 254e62d88c3242a0a3ac7470a8005d74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254e62d88c3242a0a3ac7470a8005d74 already in progress: creating tablet
I20251029 08:52:15.045819 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]) successfully deleted
I20251029 08:52:15.046125 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b11998f3ebf3402ab3f729d04fd5e0ef (table default.Table_508e306ea26d403299fafa0a0cd2419c [id=f9c1873a2560495fa17fc61a1c1906b6]) successfully deleted
W20251029 08:52:15.046324 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b49b69d8e23446a88c9e28445a8b4b56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49b69d8e23446a88c9e28445a8b4b56 already in progress: creating tablet
I20251029 08:52:15.048624 1177 tablet_bootstrap.cc:654] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.049574 1175 ts_tablet_manager.cc:1916] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.050133 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215cd666c71a4179a19e298868a88e96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:15.050793 1183 tablet_bootstrap.cc:492] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.050952 1183 ts_tablet_manager.cc:1403] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 08:52:15.051661 1183 raft_consensus.cc:359] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.051841 1183 raft_consensus.cc:385] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.051908 1183 raft_consensus.cc:740] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:15.052482 718 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.052613 718 leader_election.cc:304] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.052945 1774 raft_consensus.cc:2749] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.053556 1176 ts_tablet_manager.cc:1929] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.054772 1863 raft_consensus.cc:515] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.054816 1183 consensus_queue.cc:260] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.055058 1176 log.cc:1199] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:15.055423 1176 ts_tablet_manager.cc:1950] T ee8117e04f5f40e98ea755dc461067dd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.059368 1183 ts_tablet_manager.cc:1434] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:15.054678 1178 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.059964 1178 ts_tablet_manager.cc:1403] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.007s
I20251029 08:52:15.060738 1178 raft_consensus.cc:359] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.060950 1178 raft_consensus.cc:385] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.061035 1178 raft_consensus.cc:740] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.061286 1178 consensus_queue.cc:260] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.061775 1178 ts_tablet_manager.cc:1434] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:15.063512 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:15.065263 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
W20251029 08:52:15.065336 861 leader_election.cc:343] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.065443 782 raft_consensus.cc:3060] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:15.068485 782 raft_consensus.cc:2468] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:15.068848 861 leader_election.cc:304] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:15.069440 1822 raft_consensus.cc:2804] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:15.069589 1822 raft_consensus.cc:697] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:15.069592 1179 ts_tablet_manager.cc:1929] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:15.069746 1178 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.069904 1179 log.cc:1199] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4a756d6b5b3648e88cd2bd3400e51bed
I20251029 08:52:15.070370 1179 ts_tablet_manager.cc:1950] T 4a756d6b5b3648e88cd2bd3400e51bed P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.071254 1822 consensus_queue.cc:237] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.071583 1178 tablet_bootstrap.cc:654] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.072237 1863 leader_election.cc:290] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.073305 1179 tablet_replica.cc:333] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.073534 1179 raft_consensus.cc:2243] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.074537 1179 raft_consensus.cc:2272] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.075599 1183 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.075721 1175 ts_tablet_manager.cc:1929] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.075863 1175 log.cc:1199] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:15.077448 1183 tablet_bootstrap.cc:654] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.077664 1176 tablet_replica.cc:333] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.077889 1176 raft_consensus.cc:2243] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.079299 1176 raft_consensus.cc:2272] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.079881 1176 ts_tablet_manager.cc:1916] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.078440 1179 ts_tablet_manager.cc:1916] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.095580 1183 tablet_bootstrap.cc:492] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.095840 1183 ts_tablet_manager.cc:1403] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.001s
I20251029 08:52:15.096714 1183 raft_consensus.cc:359] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.096969 1183 raft_consensus.cc:385] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.097106 1183 raft_consensus.cc:740] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.097438 1183 consensus_queue.cc:260] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.098590 1178 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.098727 1178 ts_tablet_manager.cc:1403] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20251029 08:52:15.100050 1177 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.100178 1177 ts_tablet_manager.cc:1403] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20251029 08:52:15.100096 1178 raft_consensus.cc:359] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.100314 1178 raft_consensus.cc:385] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.100369 1178 raft_consensus.cc:740] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.100579 1178 consensus_queue.cc:260] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.100889 1177 raft_consensus.cc:359] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.101024 1178 ts_tablet_manager.cc:1434] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:15.101047 1177 raft_consensus.cc:385] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.101096 1177 raft_consensus.cc:740] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.101313 1178 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.101264 1177 consensus_queue.cc:260] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.101714 1177 ts_tablet_manager.cc:1434] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 08:52:15.101963 1177 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.103857 1183 ts_tablet_manager.cc:1434] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 08:52:15.104108 1177 tablet_bootstrap.cc:654] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.111086 1183 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.113163 1183 tablet_bootstrap.cc:654] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.116693 1179 ts_tablet_manager.cc:1929] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.116824 1179 log.cc:1199] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4deeec492a2545ca86583c6ea899a0ab
I20251029 08:52:15.117233 1179 ts_tablet_manager.cc:1950] T 4deeec492a2545ca86583c6ea899a0ab P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.049536 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f0495c28735e41e2b08d71fc162f7477" } new_table_name: "default.Table_7aa2b72bba9d4f6d80cc2a4a09825a94" modify_external_catalogs: true
I20251029 08:52:15.049933 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9f2f9d3d1fe34371b8a67eaf218a4154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.118923 1179 tablet_replica.cc:333] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:15.118947 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f2f9d3d1fe34371b8a67eaf218a4154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.119114 1179 raft_consensus.cc:2243] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.119280 1179 raft_consensus.cc:2272] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.119799 1179 ts_tablet_manager.cc:1916] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.051712 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_656e3c6b753e4109bc607af874503567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.120383 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_656e3c6b753e4109bc607af874503567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:15.082716 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1e2be77182e4259b76d20cae4e9e478 consensus state has no leader
W20251029 08:52:15.123615 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 77308.8us
I20251029 08:52:15.123960 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2e96cbdc4b7845b3823617a8e9a8c8b2 (table default.Table_b9b1c4a272214856ab50337015cdbd7b [id=4856bf7f477c47d58060d1728c544aa0]) successfully deleted
I20251029 08:52:15.124125 1178 tablet_bootstrap.cc:654] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.124408 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 254e62d88c3242a0a3ac7470a8005d74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254e62d88c3242a0a3ac7470a8005d74 already in progress: creating tablet
I20251029 08:52:15.124744 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]) successfully deleted
I20251029 08:52:15.124965 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4deeec492a2545ca86583c6ea899a0ab (table default.Table_b90c808b8f5041bbbe0c6ae3515e4a1e [id=58d71d681f8b4cd4967daf7d53eb5191]) successfully deleted
W20251029 08:52:15.125239 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 85ee033a724942b0a90f12f15cba2701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ee033a724942b0a90f12f15cba2701 already in progress: creating tablet
W20251029 08:52:15.125589 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c already in progress: creating tablet
W20251029 08:52:15.125835 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5ec6a9c863b14029b8651e45aea5104e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec6a9c863b14029b8651e45aea5104e already in progress: creating tablet
I20251029 08:52:15.125922 1176 ts_tablet_manager.cc:1929] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:15.126066 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ba201a54466240129d82a096f9316242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba201a54466240129d82a096f9316242 already in progress: creating tablet
I20251029 08:52:15.126139 1176 log.cc:1199] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/215cd666c71a4179a19e298868a88e96
I20251029 08:52:15.126331 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ee8117e04f5f40e98ea755dc461067dd (table default.Table_d6dc6ba680874a51afe94c07adb83e2b [id=5730455f82a94d8698ac62979d87d9d6]) successfully deleted
I20251029 08:52:15.126474 1176 ts_tablet_manager.cc:1950] T 215cd666c71a4179a19e298868a88e96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.133423 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]). Expected version 2 got 0
I20251029 08:52:15.134032 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 215cd666c71a4179a19e298868a88e96 (table default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]) successfully deleted
W20251029 08:52:15.134266 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 85ee033a724942b0a90f12f15cba2701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ee033a724942b0a90f12f15cba2701 already in progress: creating tablet
W20251029 08:52:15.134464 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ba201a54466240129d82a096f9316242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba201a54466240129d82a096f9316242 already in progress: creating tablet
W20251029 08:52:15.134680 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5ec6a9c863b14029b8651e45aea5104e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec6a9c863b14029b8651e45aea5104e already in progress: creating tablet
I20251029 08:52:15.135021 1175 ts_tablet_manager.cc:1950] T add0605edf5c4511ae4c8d9cb7f0babb P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.135601 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet add0605edf5c4511ae4c8d9cb7f0babb (table default.Table_3eda8a3eed924142880c9b465eb759b9 [id=edda8e15b59b472886bb1eba8fec8739]) successfully deleted
W20251029 08:52:15.136359 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c already in progress: creating tablet
W20251029 08:52:15.136686 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 24ab9df6f768410eb4f66cffdcd68df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ab9df6f768410eb4f66cffdcd68df2 already in progress: creating tablet
I20251029 08:52:15.137480 1751 raft_consensus.cc:493] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.137686 1751 raft_consensus.cc:515] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.137967 762 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.134461 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9a1d9f17776045feba0e8b4104b2fd24" } modify_external_catalogs: true
I20251029 08:52:15.138561 1751 leader_election.cc:290] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.138952 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9a1d9f17776045feba0e8b4104b2fd24" } modify_external_catalogs: true
W20251029 08:52:15.140079 996 leader_election.cc:343] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 9702a7d8b8304b49a250c129ecf28303
W20251029 08:52:15.140245 998 leader_election.cc:343] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:15.140446 998 leader_election.cc:304] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.141544 1178 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.141714 1178 ts_tablet_manager.cc:1403] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.001s
I20251029 08:52:15.142179 1176 tablet_replica.cc:333] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.142374 1176 raft_consensus.cc:2243] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.142772 1751 raft_consensus.cc:2749] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.142519 1176 raft_consensus.cc:2272] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.142577 1178 raft_consensus.cc:359] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.143221 1178 raft_consensus.cc:385] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.143287 1178 raft_consensus.cc:740] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.143585 1178 consensus_queue.cc:260] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.143867 1176 ts_tablet_manager.cc:1916] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.144659 1179 ts_tablet_manager.cc:1929] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.144827 1179 log.cc:1199] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:15.145180 1179 ts_tablet_manager.cc:1950] T 39dc5ebeb36043e0aa5955bacd06c2ef P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.146567 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 39dc5ebeb36043e0aa5955bacd06c2ef (table default.Table_57a9dba12bd24aedb38524e8c99dfe84 [id=6b038b1c9cce4b27bc2486bb5132224d]) successfully deleted
W20251029 08:52:15.146909 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 24ab9df6f768410eb4f66cffdcd68df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ab9df6f768410eb4f66cffdcd68df2 already in progress: creating tablet
I20251029 08:52:15.147637 1183 tablet_bootstrap.cc:492] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.147929 1183 ts_tablet_manager.cc:1403] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.005s
I20251029 08:52:15.148772 1183 raft_consensus.cc:359] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.149024 1183 raft_consensus.cc:385] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.149133 1183 raft_consensus.cc:740] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.151664 1779 raft_consensus.cc:493] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.149470 1183 consensus_queue.cc:260] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.151858 1779 raft_consensus.cc:515] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.152933 1779 leader_election.cc:290] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:15.153014 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 254e62d88c3242a0a3ac7470a8005d74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254e62d88c3242a0a3ac7470a8005d74 already in progress: creating tablet
I20251029 08:52:15.153483 1178 ts_tablet_manager.cc:1434] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 08:52:15.153548 1175 tablet_replica.cc:333] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:15.153730 1175 raft_consensus.cc:2243] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:15.154014 1175 raft_consensus.cc:2272] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.154363 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.154560 1175 ts_tablet_manager.cc:1916] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:15.157888 861 leader_election.cc:343] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.159480 1183 ts_tablet_manager.cc:1434] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:15.164489 1183 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.166473 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:15.166676 782 raft_consensus.cc:2468] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:15.166908 1183 tablet_bootstrap.cc:654] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.167296 861 leader_election.cc:304] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:15.167848 1779 raft_consensus.cc:2804] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:15.167953 1779 raft_consensus.cc:493] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:15.168056 1779 raft_consensus.cc:3060] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:15.171520 1779 raft_consensus.cc:515] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.172659 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:15.172847 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:15.173010 782 raft_consensus.cc:3060] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:15.173882 861 leader_election.cc:343] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.174069 1779 leader_election.cc:290] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.174299 1178 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.178074 1175 ts_tablet_manager.cc:1929] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:15.178193 1175 log.cc:1199] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4a756d6b5b3648e88cd2bd3400e51bed
I20251029 08:52:15.178560 1175 ts_tablet_manager.cc:1950] T 4a756d6b5b3648e88cd2bd3400e51bed P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.179579 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4a756d6b5b3648e88cd2bd3400e51bed (table default.Table_0f39ad7a803c4432ada5c9cf85ae0632 [id=63588e39919a481c856255fd848bbf98]) successfully deleted
I20251029 08:52:15.181475 782 raft_consensus.cc:2468] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:15.181921 1176 ts_tablet_manager.cc:1929] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.182026 1176 log.cc:1199] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:15.182343 1176 ts_tablet_manager.cc:1950] T a8c5008d9df9419bb4b6d678c6dc27a6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:15.182451 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 85ee033a724942b0a90f12f15cba2701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ee033a724942b0a90f12f15cba2701 already in progress: creating tablet
I20251029 08:52:15.183104 861 leader_election.cc:304] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:15.183598 1178 tablet_bootstrap.cc:654] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.184186 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c already in progress: creating tablet
W20251029 08:52:15.184456 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5ec6a9c863b14029b8651e45aea5104e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec6a9c863b14029b8651e45aea5104e already in progress: creating tablet
W20251029 08:52:15.184664 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ba201a54466240129d82a096f9316242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba201a54466240129d82a096f9316242 already in progress: creating tablet
W20251029 08:52:15.184904 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 24ab9df6f768410eb4f66cffdcd68df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ab9df6f768410eb4f66cffdcd68df2 already in progress: creating tablet
I20251029 08:52:15.188882 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a8c5008d9df9419bb4b6d678c6dc27a6 (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:15.200459 1779 raft_consensus.cc:2804] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:15.201198 1779 raft_consensus.cc:697] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:15.201650 1779 consensus_queue.cc:237] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.202786 1183 tablet_bootstrap.cc:492] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.202937 1183 ts_tablet_manager.cc:1403] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.011s
I20251029 08:52:15.203682 1183 raft_consensus.cc:359] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.203851 1183 raft_consensus.cc:385] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.203928 1183 raft_consensus.cc:740] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.205060 1178 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.204212 1183 consensus_queue.cc:260] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.205205 1178 ts_tablet_manager.cc:1403] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251029 08:52:15.205874 1178 raft_consensus.cc:359] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.206039 1178 raft_consensus.cc:385] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.206100 1178 raft_consensus.cc:740] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.206321 1178 consensus_queue.cc:260] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.206826 1178 ts_tablet_manager.cc:1434] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:15.207120 1178 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.208846 1178 tablet_bootstrap.cc:654] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.213999 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_40ff04cf3a70422293d9648f7364d8a0" } modify_external_catalogs: true
I20251029 08:52:15.214187 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_40ff04cf3a70422293d9648f7364d8a0" } modify_external_catalogs: true
I20251029 08:52:15.216554 1183 ts_tablet_manager.cc:1434] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 08:52:15.221222 1183 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.223724 1183 tablet_bootstrap.cc:654] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.224717 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 consensus state has no leader
I20251029 08:52:15.225394 1178 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.225576 1178 ts_tablet_manager.cc:1403] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251029 08:52:15.226588 1178 raft_consensus.cc:359] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.226815 1178 raft_consensus.cc:385] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.226887 1178 raft_consensus.cc:740] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.227157 1178 consensus_queue.cc:260] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.227788 1178 ts_tablet_manager.cc:1434] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.228302 1178 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.230048 1178 tablet_bootstrap.cc:654] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.241374 1177 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.241513 1177 ts_tablet_manager.cc:1403] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.140s user 0.018s sys 0.032s
I20251029 08:52:15.242249 1177 raft_consensus.cc:359] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.242453 1177 raft_consensus.cc:385] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.242504 1177 raft_consensus.cc:740] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.242867 1177 consensus_queue.cc:260] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.243505 1177 ts_tablet_manager.cc:1434] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.243799 1177 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.244081 1178 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.244244 1178 ts_tablet_manager.cc:1403] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20251029 08:52:15.245190 1178 raft_consensus.cc:359] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.245343 1177 tablet_bootstrap.cc:654] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.245400 1178 raft_consensus.cc:385] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.245474 1178 raft_consensus.cc:740] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.245757 1178 consensus_queue.cc:260] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.246351 1178 ts_tablet_manager.cc:1434] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:15.251737 1178 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.253872 1178 tablet_bootstrap.cc:654] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.258446 1183 tablet_bootstrap.cc:492] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.258628 1183 ts_tablet_manager.cc:1403] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.018s
I20251029 08:52:15.258869 1822 raft_consensus.cc:493] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.259001 1822 raft_consensus.cc:515] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.259608 1822 leader_election.cc:290] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.259984 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abc307b14aff4c5e937a5151aab1dfb9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.260558 861 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: abc307b14aff4c5e937a5151aab1dfb9
W20251029 08:52:15.260830 861 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.260885 1178 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.260938 861 leader_election.cc:304] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.261019 1178 ts_tablet_manager.cc:1403] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251029 08:52:15.261188 1822 raft_consensus.cc:2749] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.261245 1183 raft_consensus.cc:359] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.261647 1178 raft_consensus.cc:359] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.261783 1178 raft_consensus.cc:385] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.261824 1178 raft_consensus.cc:740] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.262001 1178 consensus_queue.cc:260] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.262408 1178 ts_tablet_manager.cc:1434] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:15.262683 1183 raft_consensus.cc:385] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.262778 1183 raft_consensus.cc:740] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.263154 1177 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.263062 1183 consensus_queue.cc:260] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.263341 1177 ts_tablet_manager.cc:1403] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20251029 08:52:15.263567 1183 ts_tablet_manager.cc:1434] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:15.264081 1177 raft_consensus.cc:359] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.264295 1183 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.264285 1177 raft_consensus.cc:385] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.264405 1177 raft_consensus.cc:740] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.264648 1177 consensus_queue.cc:260] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.265250 1177 ts_tablet_manager.cc:1434] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:15.265580 1177 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.265678 1178 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.266014 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8360e68864e74c0b8acca762339062b4" } modify_external_catalogs: true
I20251029 08:52:15.266232 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8360e68864e74c0b8acca762339062b4" } modify_external_catalogs: true
I20251029 08:52:15.267360 1177 tablet_bootstrap.cc:654] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.267390 1178 tablet_bootstrap.cc:654] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.275523 1183 tablet_bootstrap.cc:654] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.278538 1178 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.278687 1178 ts_tablet_manager.cc:1403] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20251029 08:52:15.279482 1178 raft_consensus.cc:359] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.279647 1178 raft_consensus.cc:385] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.279701 1178 raft_consensus.cc:740] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.279893 1178 consensus_queue.cc:260] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.280426 1178 ts_tablet_manager.cc:1434] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:15.280849 1178 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.282594 1178 tablet_bootstrap.cc:654] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.285921 1177 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.286070 1177 ts_tablet_manager.cc:1403] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20251029 08:52:15.286835 1177 raft_consensus.cc:359] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.287021 1177 raft_consensus.cc:385] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.287086 1177 raft_consensus.cc:740] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.287370 1177 consensus_queue.cc:260] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.287938 1177 ts_tablet_manager.cc:1434] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:15.288231 1177 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.289784 1177 tablet_bootstrap.cc:654] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.292898 1183 tablet_bootstrap.cc:492] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.293108 1183 ts_tablet_manager.cc:1403] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.008s
I20251029 08:52:15.294163 1183 raft_consensus.cc:359] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.294405 1183 raft_consensus.cc:385] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.294512 1183 raft_consensus.cc:740] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.294791 1183 consensus_queue.cc:260] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.295455 1183 ts_tablet_manager.cc:1434] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:15.297506 1183 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.299273 1183 tablet_bootstrap.cc:654] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.303697 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 49713.4us
I20251029 08:52:15.322669 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ee1b6420e8a74932826b5a476ee2d9b2" } new_table_name: "default.Table_4ca13bdf97f04056ba795f885976e736" modify_external_catalogs: true
I20251029 08:52:15.343299 1751 raft_consensus.cc:493] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.343545 1751 raft_consensus.cc:515] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.344281 1751 leader_election.cc:290] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.345510 1178 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.345643 1178 ts_tablet_manager.cc:1403] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.025s
I20251029 08:52:15.346382 1178 raft_consensus.cc:359] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.346544 1178 raft_consensus.cc:385] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:15.346508 998 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 28face87091b49d2971217618c95d2f5
I20251029 08:52:15.346607 1178 raft_consensus.cc:740] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.346841 1178 consensus_queue.cc:260] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:15.346902 996 leader_election.cc:343] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 28face87091b49d2971217618c95d2f5
I20251029 08:52:15.347035 996 leader_election.cc:304] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.347308 1178 ts_tablet_manager.cc:1434] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.348289 1751 raft_consensus.cc:2749] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.349352 1177 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.349500 1177 ts_tablet_manager.cc:1403] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.020s
I20251029 08:52:15.349756 1178 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.350244 1177 raft_consensus.cc:359] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.350418 1177 raft_consensus.cc:385] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.350471 1177 raft_consensus.cc:740] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.350664 1177 consensus_queue.cc:260] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.351330 1178 tablet_bootstrap.cc:654] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.356494 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_87bd0ffa8ae04a5da1b10cae0b440517" } new_table_name: "default.Table_4151e370ccfb49d9b6cf20513af24e60" modify_external_catalogs: true
I20251029 08:52:15.360940 1178 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.361092 1178 ts_tablet_manager.cc:1403] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20251029 08:52:15.361856 1178 raft_consensus.cc:359] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.362040 1178 raft_consensus.cc:385] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.362110 1178 raft_consensus.cc:740] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.362351 1178 consensus_queue.cc:260] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.362798 1178 ts_tablet_manager.cc:1434] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.364269 1178 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.364374 1177 ts_tablet_manager.cc:1434] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 08:52:15.364758 1177 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.365860 1183 tablet_bootstrap.cc:492] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.365896 1178 tablet_bootstrap.cc:654] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.366053 1183 ts_tablet_manager.cc:1403] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.004s
I20251029 08:52:15.366837 1183 raft_consensus.cc:359] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.367156 1183 raft_consensus.cc:385] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.367238 1183 raft_consensus.cc:740] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.367508 1183 consensus_queue.cc:260] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.370286 1183 ts_tablet_manager.cc:1434] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:15.370626 1183 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.372264 1183 tablet_bootstrap.cc:654] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.374097 1177 tablet_bootstrap.cc:654] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.377990 1183 tablet_bootstrap.cc:492] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.378139 1183 ts_tablet_manager.cc:1403] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20251029 08:52:15.378276 1178 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.378415 1178 ts_tablet_manager.cc:1403] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20251029 08:52:15.379343 1178 raft_consensus.cc:359] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.379510 1178 raft_consensus.cc:385] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.379441 1183 raft_consensus.cc:359] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.379571 1178 raft_consensus.cc:740] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.379618 1183 raft_consensus.cc:385] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.379709 1183 raft_consensus.cc:740] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.379801 1178 consensus_queue.cc:260] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.379918 1183 consensus_queue.cc:260] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.380290 1178 ts_tablet_manager.cc:1434] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:15.380549 1178 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.381415 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc] alter complete (version 2)
I20251029 08:52:15.382118 1178 tablet_bootstrap.cc:654] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.382488 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dec33b502aa4916b89b878462de1ab1 consensus state has no leader
I20251029 08:52:15.383152 1183 ts_tablet_manager.cc:1434] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 08:52:15.386003 1183 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.388363 1183 tablet_bootstrap.cc:654] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.388904 1178 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.389045 1178 ts_tablet_manager.cc:1403] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20251029 08:52:15.389374 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c253bd6532c8472295cf1d50dfc59d42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.389811 1178 raft_consensus.cc:359] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.389964 1178 raft_consensus.cc:385] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:15.389976 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c253bd6532c8472295cf1d50dfc59d42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.390029 1178 raft_consensus.cc:740] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.390237 1178 consensus_queue.cc:260] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.390710 1178 ts_tablet_manager.cc:1434] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:15.390985 1178 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.392604 1178 tablet_bootstrap.cc:654] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.393898 1183 tablet_bootstrap.cc:492] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.394065 1183 ts_tablet_manager.cc:1403] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
W20251029 08:52:15.394404 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8721f0ab79d64b24a7b8b4797a5a3f2b consensus state has no leader
I20251029 08:52:15.394809 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8d2ea6ac0509441282fd5b4a11217079" } new_table_name: "default.Table_a97711da23ff4ab79b7f1810e23031c6" modify_external_catalogs: true
I20251029 08:52:15.394855 1183 raft_consensus.cc:359] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.395041 1183 raft_consensus.cc:385] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.395131 1183 raft_consensus.cc:740] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.395383 1183 consensus_queue.cc:260] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.398924 1183 ts_tablet_manager.cc:1434] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:15.401923 1183 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.403710 1183 tablet_bootstrap.cc:654] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.405416 1177 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.405467 1178 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.405640 1178 ts_tablet_manager.cc:1403] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251029 08:52:15.405854 1177 ts_tablet_manager.cc:1403] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20251029 08:52:15.406546 1178 raft_consensus.cc:359] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.406810 1178 raft_consensus.cc:385] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.406921 1178 raft_consensus.cc:740] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.407004 1177 raft_consensus.cc:359] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.407227 1177 raft_consensus.cc:385] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.407298 1177 raft_consensus.cc:740] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.407341 1178 consensus_queue.cc:260] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.407586 1177 consensus_queue.cc:260] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.408114 1178 ts_tablet_manager.cc:1434] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:15.408414 1178 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.410257 1177 ts_tablet_manager.cc:1434] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:15.410521 1177 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.411450 1178 tablet_bootstrap.cc:654] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.412230 1177 tablet_bootstrap.cc:654] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.415472 1183 tablet_bootstrap.cc:492] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:15.415617 1183 ts_tablet_manager.cc:1403] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20251029 08:52:15.416378 1183 raft_consensus.cc:359] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.416560 1183 raft_consensus.cc:385] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.416656 1183 raft_consensus.cc:740] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.416880 1183 consensus_queue.cc:260] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.417493 1183 ts_tablet_manager.cc:1434] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:15.417757 1183 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:15.419965 1183 tablet_bootstrap.cc:654] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.423827 1774 raft_consensus.cc:493] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.424000 1774 raft_consensus.cc:515] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.424664 1774 leader_election.cc:290] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.425412 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba1bcb3f6c134de2b322696501927a16" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.425993 720 leader_election.cc:343] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.426683 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba1bcb3f6c134de2b322696501927a16" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.427855 718 leader_election.cc:343] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.427964 718 leader_election.cc:304] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.428426 1774 raft_consensus.cc:2749] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.430106 1774 raft_consensus.cc:493] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.430275 1774 raft_consensus.cc:515] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.430990 1774 leader_election.cc:290] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.431533 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff4e625abf14fbd94a8102dfa5fc993" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.431944 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff4e625abf14fbd94a8102dfa5fc993" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.432564 720 leader_election.cc:343] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:15.433374 718 leader_election.cc:343] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.433473 718 leader_election.cc:304] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.433887 1774 raft_consensus.cc:2749] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.439987 1774 raft_consensus.cc:493] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.440140 1774 raft_consensus.cc:515] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.440734 1774 leader_election.cc:290] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.441531 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b665b63a7341d699d499ffc8cfd9f7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.442152 720 leader_election.cc:343] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.442628 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b665b63a7341d699d499ffc8cfd9f7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.443154 718 leader_election.cc:343] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.443256 718 leader_election.cc:304] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.443650 1774 raft_consensus.cc:2749] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.444523 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_d4b6d93a144541b3b24624df6e5a0201 [id=821160def5624241afdce0f734ef2411] alter complete (version 3)
I20251029 08:52:15.445247 1774 raft_consensus.cc:493] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.445413 1774 raft_consensus.cc:515] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.446164 1774 leader_election.cc:290] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.447029 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a31847928db44bd4b1ebd50c5130065f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.447403 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a31847928db44bd4b1ebd50c5130065f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.448361 718 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:15.448751 720 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.448844 720 leader_election.cc:304] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.449250 1774 raft_consensus.cc:2749] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.453047 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:15.453863 595 catalog_manager.cc:5649] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:15.455772 1774 raft_consensus.cc:493] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.455880 1774 raft_consensus.cc:515] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.456545 1774 leader_election.cc:290] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.457306 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a7a673ba4fd4abcbfc3493c38dcc257" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.457643 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:15.458146 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a7a673ba4fd4abcbfc3493c38dcc257" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:15.458022 595 catalog_manager.cc:5649] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
W20251029 08:52:15.458884 718 leader_election.cc:343] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.458976 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
W20251029 08:52:15.459348 720 leader_election.cc:343] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.459489 720 leader_election.cc:304] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.460099 1774 raft_consensus.cc:2749] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.460564 591 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_4028cb8d8c3e48c48ef5c82e7febdd54 [id=a23a597c8fd343769b517507845104ff] alter complete (version 1)
I20251029 08:52:15.464800 1751 raft_consensus.cc:493] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.464988 1751 raft_consensus.cc:515] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.465068 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 39dc5ebeb36043e0aa5955bacd06c2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.465175 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 983a406aeed243f29ab322f94a0f5239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.465329 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 9702a7d8b8304b49a250c129ecf28303 with delete_type TABLET_DATA_DELETED (replaced by tablet 7be4bcf9e3324bd884e946fae20648fd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.465631 1751 leader_election.cc:290] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.465718 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.465922 1176 tablet_replica.cc:333] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.466161 1176 raft_consensus.cc:2243] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.466321 1176 raft_consensus.cc:2272] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.465924 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.467839 1176 ts_tablet_manager.cc:1916] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.467933 1179 tablet_replica.cc:333] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.468143 1179 raft_consensus.cc:2243] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.468333 1179 raft_consensus.cc:2272] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.468873 1179 ts_tablet_manager.cc:1916] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.470532 762 tablet_service.cc:1505] Processing CreateTablet for tablet 71bedce60cd34585adfed95e72ab6cd0 (DEFAULT_TABLE table=default.Table_a705dde62dcc4a949981ec95d2b75a98 [id=e96842ffb8f1430bbcc4496c4a533986]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.470954 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bedce60cd34585adfed95e72ab6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.472082 895 tablet_service.cc:1505] Processing CreateTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf (DEFAULT_TABLE table=default.Table_fc8d5092b4774b28adcf5f8a06dca48d [id=f4d7195a8ac4492691413a0dee606d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.472564 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c3357ebff4dd08df4ed2477b40ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.473404 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 71bedce60cd34585adfed95e72ab6cd0 (DEFAULT_TABLE table=default.Table_a705dde62dcc4a949981ec95d2b75a98 [id=e96842ffb8f1430bbcc4496c4a533986]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:15.473618 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 39dc5ebeb36043e0aa5955bacd06c2ef because the tablet was not found. No further retry: Not found: Tablet not found: 39dc5ebeb36043e0aa5955bacd06c2ef
I20251029 08:52:15.473798 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:15.474045 782 raft_consensus.cc:2393] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:15.474440 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bedce60cd34585adfed95e72ab6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.476883 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf (DEFAULT_TABLE table=default.Table_fc8d5092b4774b28adcf5f8a06dca48d [id=f4d7195a8ac4492691413a0dee606d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.477421 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c3357ebff4dd08df4ed2477b40ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.478227 1822 raft_consensus.cc:493] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.478410 1822 raft_consensus.cc:515] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.478983 1822 leader_election.cc:290] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.479097 1822 raft_consensus.cc:493] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.479192 1822 raft_consensus.cc:515] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.479666 1822 leader_election.cc:290] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.479624 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10baa0547180493f872a23614cf4c608" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:15.480131 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa8884bb8e64474d936c8825d1ef3545" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.480528 861 leader_election.cc:343] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fa8884bb8e64474d936c8825d1ef3545
W20251029 08:52:15.480742 861 leader_election.cc:343] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.480819 861 leader_election.cc:304] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.480844 1179 ts_tablet_manager.cc:1929] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.480947 1179 log.cc:1199] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/983a406aeed243f29ab322f94a0f5239
I20251029 08:52:15.481056 1822 raft_consensus.cc:2749] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.481580 1179 ts_tablet_manager.cc:1950] T 983a406aeed243f29ab322f94a0f5239 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.482576 761 tablet_service.cc:1505] Processing CreateTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf (DEFAULT_TABLE table=default.Table_fc8d5092b4774b28adcf5f8a06dca48d [id=f4d7195a8ac4492691413a0dee606d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.482671 1179 tablet_replica.cc:333] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.482867 1179 raft_consensus.cc:2243] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.483007 1179 raft_consensus.cc:2272] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.483054 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c3357ebff4dd08df4ed2477b40ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.483520 1179 ts_tablet_manager.cc:1916] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.487805 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dee931aa035748d08b1260e8ab3fca5c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.487942 861 leader_election.cc:343] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: dee931aa035748d08b1260e8ab3fca5c
I20251029 08:52:15.488888 996 leader_election.cc:304] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.489450 1751 raft_consensus.cc:3060] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:15.492151 861 leader_election.cc:343] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.492254 861 leader_election.cc:304] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.492323 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 983a406aeed243f29ab322f94a0f5239 (table default.Table_a09dfe0f96c64010a72526b0f6cb3920 [id=4a92da4eb5774d53895200ae074754a0]) successfully deleted
I20251029 08:52:15.492604 1779 raft_consensus.cc:2749] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.493021 1179 ts_tablet_manager.cc:1929] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.493152 1179 log.cc:1199] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9702a7d8b8304b49a250c129ecf28303
I20251029 08:52:15.493587 1179 ts_tablet_manager.cc:1950] T 9702a7d8b8304b49a250c129ecf28303 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.494607 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9702a7d8b8304b49a250c129ecf28303 (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
I20251029 08:52:15.495348 1751 raft_consensus.cc:2749] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.497967 896 tablet_service.cc:1505] Processing CreateTablet for tablet 71bedce60cd34585adfed95e72ab6cd0 (DEFAULT_TABLE table=default.Table_a705dde62dcc4a949981ec95d2b75a98 [id=e96842ffb8f1430bbcc4496c4a533986]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.498486 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bedce60cd34585adfed95e72ab6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.506572 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 0b44da3f8d684a10ae8f1c9b856d3c64 (DEFAULT_TABLE table=default.Table_0c27f6be2f8f414bb78ef1ffb99d8e60 [id=b825ff06ea204a35b4e92bc4289c12aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.506870 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0b44da3f8d684a10ae8f1c9b856d3c64 (DEFAULT_TABLE table=default.Table_0c27f6be2f8f414bb78ef1ffb99d8e60 [id=b825ff06ea204a35b4e92bc4289c12aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.506968 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b44da3f8d684a10ae8f1c9b856d3c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.506880 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0b44da3f8d684a10ae8f1c9b856d3c64 (DEFAULT_TABLE table=default.Table_0c27f6be2f8f414bb78ef1ffb99d8e60 [id=b825ff06ea204a35b4e92bc4289c12aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.507189 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b44da3f8d684a10ae8f1c9b856d3c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.507383 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b44da3f8d684a10ae8f1c9b856d3c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.521114 1049 tablet_service.cc:1505] Processing CreateTablet for tablet f49ec4f21035419c9f2cde4b26731f32 (DEFAULT_TABLE table=default.Table_77649b9af0074613a28f9084f9f924d7 [id=cac7a27553ae4001a3a8a4d1c682d91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.521601 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49ec4f21035419c9f2cde4b26731f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.523072 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 039ecc2142964a3782f6879b53f3b73e (DEFAULT_TABLE table=default.Table_c90eeb172173429785cab8c6674fbfcc [id=38c8744eaccd48bab5e40e309c5763b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.523573 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039ecc2142964a3782f6879b53f3b73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.525408 762 tablet_service.cc:1505] Processing CreateTablet for tablet f49ec4f21035419c9f2cde4b26731f32 (DEFAULT_TABLE table=default.Table_77649b9af0074613a28f9084f9f924d7 [id=cac7a27553ae4001a3a8a4d1c682d91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.525792 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49ec4f21035419c9f2cde4b26731f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.528303 896 tablet_service.cc:1505] Processing CreateTablet for tablet f49ec4f21035419c9f2cde4b26731f32 (DEFAULT_TABLE table=default.Table_77649b9af0074613a28f9084f9f924d7 [id=cac7a27553ae4001a3a8a4d1c682d91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.528342 759 tablet_service.cc:1505] Processing CreateTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f (DEFAULT_TABLE table=default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.528658 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49ec4f21035419c9f2cde4b26731f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.528812 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf09e8fd3d2b43318fa37d922c8b826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.532346 1035 tablet_service.cc:1505] Processing CreateTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f (DEFAULT_TABLE table=default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.532838 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf09e8fd3d2b43318fa37d922c8b826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.536863 1177 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.537009 1177 ts_tablet_manager.cc:1403] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.127s user 0.002s sys 0.003s
I20251029 08:52:15.536917 902 tablet_service.cc:1505] Processing CreateTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f (DEFAULT_TABLE table=default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.537400 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf09e8fd3d2b43318fa37d922c8b826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.540544 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4876937bd3244d43b4154fdd5eb81de7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.541402 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4876937bd3244d43b4154fdd5eb81de7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.543288 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_aa5dc97a2e214cc2bb4aeaeb28706945"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.543989 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa5dc97a2e214cc2bb4aeaeb28706945 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:15.544430 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bd490da0b0e47fc965343c123a27b22 consensus state has no leader
I20251029 08:52:15.548171 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:15.548170 888 tablet_service.cc:1505] Processing CreateTablet for tablet 039ecc2142964a3782f6879b53f3b73e (DEFAULT_TABLE table=default.Table_c90eeb172173429785cab8c6674fbfcc [id=38c8744eaccd48bab5e40e309c5763b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.548548 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039ecc2142964a3782f6879b53f3b73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.548926 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:15.551909 763 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.552690 763 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.553095 763 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.553754 763 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.554380 763 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.555161 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.555244 763 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.556795 763 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:15.557137 861 consensus_peers.cc:597] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:15.558065 1176 ts_tablet_manager.cc:1929] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.557904 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_79ee6c7183a04b5c93eb9a318008b975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.558379 1176 log.cc:1199] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0ff4e625abf14fbd94a8102dfa5fc993
I20251029 08:52:15.558792 1176 ts_tablet_manager.cc:1950] T 0ff4e625abf14fbd94a8102dfa5fc993 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.559154 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.561123 1177 raft_consensus.cc:359] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.561298 1177 raft_consensus.cc:385] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.561360 1177 raft_consensus.cc:740] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:15.561338 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79ee6c7183a04b5c93eb9a318008b975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.561702 1177 consensus_queue.cc:260] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.562563 751 tablet_service.cc:1505] Processing CreateTablet for tablet 039ecc2142964a3782f6879b53f3b73e (DEFAULT_TABLE table=default.Table_c90eeb172173429785cab8c6674fbfcc [id=38c8744eaccd48bab5e40e309c5763b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.566737 1176 tablet_replica.cc:333] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.566743 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:15.567270 1176 raft_consensus.cc:2243] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.567451 1176 raft_consensus.cc:2272] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.567076 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039ecc2142964a3782f6879b53f3b73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.568182 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 09968d93203140fa9201928c87702ee0
I20251029 08:52:15.569041 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_680fa9c86aeb41c7aaa9bc8937675362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.569420 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ec00cd64dc247da8442b6ba197f370a from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:15.569581 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680fa9c86aeb41c7aaa9bc8937675362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.572912 1177 ts_tablet_manager.cc:1434] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251029 08:52:15.572961 1178 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.570850 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0ff4e625abf14fbd94a8102dfa5fc993 (table default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]) successfully deleted
I20251029 08:52:15.573087 1178 ts_tablet_manager.cc:1403] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.015s
I20251029 08:52:15.573237 1177 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.570065 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_aae83ce4ba7f4a1288306bf92745e84e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.573854 1178 raft_consensus.cc:359] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.574060 1178 raft_consensus.cc:385] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.574162 1178 raft_consensus.cc:740] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.574385 1178 consensus_queue.cc:260] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:15.574694 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aae83ce4ba7f4a1288306bf92745e84e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.576186 1178 ts_tablet_manager.cc:1434] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:15.577373 761 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.577605 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.578892 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.579339 1178 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.579437 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:15.579579 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 09968d93203140fa9201928c87702ee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09968d93203140fa9201928c87702ee0 already in progress: creating tablet
W20251029 08:52:15.580121 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 09968d93203140fa9201928c87702ee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09968d93203140fa9201928c87702ee0 already in progress: creating tablet
I20251029 08:52:15.580730 1176 ts_tablet_manager.cc:1916] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.581221 1779 consensus_queue.cc:1048] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:15.585834 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.586937 1051 tablet_service.cc:1505] Processing CreateTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 (DEFAULT_TABLE table=default.Table_2184d67d2edc408f939c2b4114a7eb15 [id=453c5eb00fb344008374ed2b3a333029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.587369 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3b6fa45044c3fb83823ed4f233735. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:15.588289 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d3955ff79faf42bb82f764aae79b85d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3955ff79faf42bb82f764aae79b85d9 already in progress: creating tablet
I20251029 08:52:15.590337 1177 tablet_bootstrap.cc:654] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.590538 888 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.590960 1178 tablet_bootstrap.cc:654] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:15.591846 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d3955ff79faf42bb82f764aae79b85d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3955ff79faf42bb82f764aae79b85d9 already in progress: creating tablet
I20251029 08:52:15.596307 1751 raft_consensus.cc:493] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.596457 1751 raft_consensus.cc:515] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.597019 1751 leader_election.cc:290] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:15.598433 996 leader_election.cc:343] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:15.607048 1176 ts_tablet_manager.cc:1929] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.607173 1176 log.cc:1199] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/14b665b63a7341d699d499ffc8cfd9f7
I20251029 08:52:15.607725 1176 ts_tablet_manager.cc:1950] T 14b665b63a7341d699d499ffc8cfd9f7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.609930 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f4c1b1d8cf7d464ebad3afca1e31258f
W20251029 08:52:15.611212 998 leader_election.cc:343] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:15.611323 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:15.611348 998 leader_election.cc:304] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.611927 1751 raft_consensus.cc:2749] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.612154 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9
I20251029 08:52:15.612423 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:15.613180 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 787d6d7f1870450fa21cb9cba34e72fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 787d6d7f1870450fa21cb9cba34e72fe already in progress: creating tablet
I20251029 08:52:15.613220 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.613471 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.613706 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.613967 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:15.614051 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 already in progress: creating tablet
W20251029 08:52:15.614645 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 787d6d7f1870450fa21cb9cba34e72fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 787d6d7f1870450fa21cb9cba34e72fe already in progress: creating tablet
I20251029 08:52:15.614972 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.615231 888 tablet_service.cc:1552] Processing DeleteTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:15.615641 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f4c1b1d8cf7d464ebad3afca1e31258f because the tablet was not found. No further retry: Not found: Tablet not found: f4c1b1d8cf7d464ebad3afca1e31258f
W20251029 08:52:15.615819 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f4c1b1d8cf7d464ebad3afca1e31258f because the tablet was not found. No further retry: Not found: Tablet not found: f4c1b1d8cf7d464ebad3afca1e31258f
I20251029 08:52:15.616207 751 tablet_service.cc:1552] Processing DeleteTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.617350 1177 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.617604 591 master_service.cc:954] ReplaceTablet: received request to replace tablet a20f26d0f7874ccb960ffe4d27e854e1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:15.617508 1177 ts_tablet_manager.cc:1403] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.001s
I20251029 08:52:15.618516 1177 raft_consensus.cc:359] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.618678 1177 raft_consensus.cc:385] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.618721 1177 raft_consensus.cc:740] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.620257 1177 consensus_queue.cc:260] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.620440 888 tablet_service.cc:1552] Processing DeleteTablet for tablet be706771a95e45e5a20a15614eadfe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.620757 1175 tablet_replica.cc:333] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:15.620932 1175 raft_consensus.cc:2243] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.621137 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.621409 888 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.621896 888 tablet_service.cc:1552] Processing DeleteTablet for tablet fa8884bb8e64474d936c8825d1ef3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.622457 1178 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.622684 1178 ts_tablet_manager.cc:1403] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.000s
I20251029 08:52:15.623713 1178 raft_consensus.cc:359] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.623992 1178 raft_consensus.cc:385] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.624146 1178 raft_consensus.cc:740] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.624686 597 master_service.cc:954] ReplaceTablet: received request to replace tablet a20f26d0f7874ccb960ffe4d27e854e1 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:15.626446 1175 raft_consensus.cc:2272] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.626981 1175 ts_tablet_manager.cc:1916] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.627202 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 14b665b63a7341d699d499ffc8cfd9f7 (table default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]) successfully deleted
I20251029 08:52:15.627586 1176 tablet_replica.cc:333] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.632570 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.635876 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
I20251029 08:52:15.636888 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_acdad95817bc4aaa86670404d517c37d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]). Expected version 2 got 0
I20251029 08:52:15.637490 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
I20251029 08:52:15.638329 1751 raft_consensus.cc:493] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.638504 1751 raft_consensus.cc:515] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.639194 1751 leader_election.cc:290] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:15.640547 998 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 23b534aaa51b4fad9dc3fc915de947f1
W20251029 08:52:15.640944 996 leader_election.cc:343] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:15.641072 996 leader_election.cc:304] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.641621 1751 raft_consensus.cc:2749] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.641827 1176 raft_consensus.cc:2243] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.642036 1176 raft_consensus.cc:2272] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.642964 1176 ts_tablet_manager.cc:1916] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.645710 1175 ts_tablet_manager.cc:1929] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.645991 1175 log.cc:1199] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/be706771a95e45e5a20a15614eadfe1a
I20251029 08:52:15.646430 1175 ts_tablet_manager.cc:1950] T be706771a95e45e5a20a15614eadfe1a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.646543 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e67ddf5ceb148dfad8f7be9f26cff64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.646852 1179 tablet_replica.cc:333] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.647107 1179 raft_consensus.cc:2243] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.647306 1179 raft_consensus.cc:2272] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.647254 751 tablet_service.cc:1505] Processing CreateTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 (DEFAULT_TABLE table=default.Table_2184d67d2edc408f939c2b4114a7eb15 [id=453c5eb00fb344008374ed2b3a333029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.647627 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3b6fa45044c3fb83823ed4f233735. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.647938 1179 ts_tablet_manager.cc:1916] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.648290 902 tablet_service.cc:1505] Processing CreateTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 (DEFAULT_TABLE table=default.Table_2184d67d2edc408f939c2b4114a7eb15 [id=453c5eb00fb344008374ed2b3a333029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.648626 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b3b6fa45044c3fb83823ed4f233735. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.650315 1176 ts_tablet_manager.cc:1929] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.650409 1176 log.cc:1199] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:15.650676 1176 ts_tablet_manager.cc:1950] T a31847928db44bd4b1ebd50c5130065f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.651997 1176 tablet_replica.cc:333] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.652138 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a31847928db44bd4b1ebd50c5130065f (table default.Table_159d7048427049ffb8b14f34b6bb78fa [id=fd07f1c36584477caf673f91f2a86f58]) successfully deleted
I20251029 08:52:15.652192 1176 raft_consensus.cc:2243] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.652352 1176 raft_consensus.cc:2272] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.652616 1774 raft_consensus.cc:493] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.652735 1774 raft_consensus.cc:515] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.653304 1774 leader_election.cc:290] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.653452 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 94738d48801a4fb4a7ad1f5c4022afe1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2723cdd513ee4e8ca6773b38f785b09e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.653846 1176 ts_tablet_manager.cc:1916] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.654058 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.654625 720 leader_election.cc:343] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.654682 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:15.655030 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 297945c50119432a9f9d3e356d573879 (DEFAULT_TABLE table=default.Table_a3f6289022a94c0393bf7f32fb2439d4 [id=426eb56052e74c0583fe6414ffcda786]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.655426 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297945c50119432a9f9d3e356d573879. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:15.655400 718 leader_election.cc:343] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.655494 718 leader_election.cc:304] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.655957 1774 raft_consensus.cc:2749] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.656230 1822 raft_consensus.cc:493] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.656369 1822 raft_consensus.cc:515] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.656925 1822 leader_election.cc:290] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.657958 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74fe974a368a4afe8a5da6dfd1036400" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.658460 861 leader_election.cc:343] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.659247 1179 ts_tablet_manager.cc:1929] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.659370 1179 log.cc:1199] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2e67ddf5ceb148dfad8f7be9f26cff64
I20251029 08:52:15.659816 1179 ts_tablet_manager.cc:1950] T 2e67ddf5ceb148dfad8f7be9f26cff64 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:15.659888 861 leader_election.cc:343] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 74fe974a368a4afe8a5da6dfd1036400
I20251029 08:52:15.660037 861 leader_election.cc:304] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.660405 1822 raft_consensus.cc:2749] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.661055 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet be706771a95e45e5a20a15614eadfe1a (table default.Table_a69aba63f41c4596a1017f38f6ed6532 [id=e20293dc36d54acf9ee6f9bf7ac55834]) successfully deleted
I20251029 08:52:15.661540 1177 ts_tablet_manager.cc:1434] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20251029 08:52:15.661778 1175 tablet_replica.cc:333] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:15.661921 1175 raft_consensus.cc:2243] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.662050 1175 raft_consensus.cc:2272] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.662528 1175 ts_tablet_manager.cc:1916] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.662688 1751 raft_consensus.cc:493] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.662807 1751 raft_consensus.cc:515] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.663393 1751 leader_election.cc:290] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.663568 1179 tablet_replica.cc:333] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.663769 1179 raft_consensus.cc:2243] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.663743 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:15.664033 780 raft_consensus.cc:2393] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:15.624470 1178 consensus_queue.cc:260] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.664422 1822 raft_consensus.cc:493] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.664564 1822 raft_consensus.cc:515] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.665203 1822 leader_election.cc:290] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.665793 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79a3ef8389944c32bba7b3e598824118" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.666224 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc95257c937454aa6cd55b5a9394a56" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.666287 861 leader_election.cc:343] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:15.666491 861 leader_election.cc:343] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 79a3ef8389944c32bba7b3e598824118
I20251029 08:52:15.666580 861 leader_election.cc:304] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.666711 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2e67ddf5ceb148dfad8f7be9f26cff64 (table default.Table_ec9165a4204e4f209e0a28ea2e0c04fe [id=7e3eb51dc35f47a497e08d2e6e62703c]) successfully deleted
I20251029 08:52:15.666781 1822 raft_consensus.cc:2749] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.666973 996 leader_election.cc:304] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.669862 1178 ts_tablet_manager.cc:1434] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.047s user 0.016s sys 0.000s
I20251029 08:52:15.670176 1178 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.672386 1178 tablet_bootstrap.cc:654] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.676218 1176 ts_tablet_manager.cc:1929] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.676433 888 tablet_service.cc:1505] Processing CreateTablet for tablet 297945c50119432a9f9d3e356d573879 (DEFAULT_TABLE table=default.Table_a3f6289022a94c0393bf7f32fb2439d4 [id=426eb56052e74c0583fe6414ffcda786]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.676872 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297945c50119432a9f9d3e356d573879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.677363 1177 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.678079 1176 log.cc:1199] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:15.678368 1176 ts_tablet_manager.cc:1950] T 7c3e50018f414f4b88e0e5d90cb7c925 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.678884 1177 tablet_bootstrap.cc:654] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.679426 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7c3e50018f414f4b88e0e5d90cb7c925 (table default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]) successfully deleted
I20251029 08:52:15.680311 751 tablet_service.cc:1505] Processing CreateTablet for tablet 297945c50119432a9f9d3e356d573879 (DEFAULT_TABLE table=default.Table_a3f6289022a94c0393bf7f32fb2439d4 [id=426eb56052e74c0583fe6414ffcda786]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.680683 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297945c50119432a9f9d3e356d573879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.680892 1176 tablet_replica.cc:333] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.681062 1176 raft_consensus.cc:2243] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.680930 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b534aaa51b4fad9dc3fc915de947f1 with delete_type TABLET_DATA_DELETED (replaced by tablet fc04616301354cfe873c825a9f5d7ab6) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.681197 1176 raft_consensus.cc:2272] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.681403 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 63b08840d0fa40428c61a05b277ad750 with delete_type TABLET_DATA_DELETED (replaced by tablet b7da7e08b74f4a9aacedada42e839a11) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.681641 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 9165c4bf484a48b4a0b7246cb11571bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.681829 1751 raft_consensus.cc:3060] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:15.684353 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 28face87091b49d2971217618c95d2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.684612 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet be706771a95e45e5a20a15614eadfe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.684782 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cbce5d595364b4bad36d59fb532c237 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:04 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.684947 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet add0605edf5c4511ae4c8d9cb7f0babb with delete_type TABLET_DATA_DELETED (replaced by tablet d26ff78aa5f94ae49a197daf2be00aa5) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.685374 1751 raft_consensus.cc:2749] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.685676 1179 raft_consensus.cc:2272] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.688704 1179 ts_tablet_manager.cc:1916] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.693004 1176 ts_tablet_manager.cc:1916] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.699015 1176 ts_tablet_manager.cc:1929] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.699126 1176 log.cc:1199] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:15.699467 1176 ts_tablet_manager.cc:1950] T ba1bcb3f6c134de2b322696501927a16 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.701159 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 300a1a816a5347efbcae9589f1cf0c61 (DEFAULT_TABLE table=default.Table_ac5705ab49ff41f5ac459ed7fe561419 [id=adfe4ac4997f40a684db06ec3f347921]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.701994 1178 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.702023 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300a1a816a5347efbcae9589f1cf0c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.702116 1178 ts_tablet_manager.cc:1403] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20251029 08:52:15.702729 1178 raft_consensus.cc:359] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.702898 1178 raft_consensus.cc:385] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.702951 1178 raft_consensus.cc:740] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.703161 1178 consensus_queue.cc:260] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.705018 1178 ts_tablet_manager.cc:1434] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:15.705283 1178 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.707185 1178 tablet_bootstrap.cc:654] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.708130 1176 tablet_replica.cc:333] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.708304 1176 raft_consensus.cc:2243] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.708452 1176 raft_consensus.cc:2272] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.708855 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ba1bcb3f6c134de2b322696501927a16 (table default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]) successfully deleted
I20251029 08:52:15.709152 1176 ts_tablet_manager.cc:1916] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.712945 1178 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.713076 1178 ts_tablet_manager.cc:1403] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20251029 08:52:15.713800 1178 raft_consensus.cc:359] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.713958 1178 raft_consensus.cc:385] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.714010 1178 raft_consensus.cc:740] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.714211 1178 consensus_queue.cc:260] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.714766 1178 ts_tablet_manager.cc:1434] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.715703 1176 ts_tablet_manager.cc:1929] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.715798 1176 log.cc:1199] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ef0a7b31ced24388bc9d9964a42fa203
I20251029 08:52:15.715766 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6a0b95dd9d0341d697d8709cd5e42be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.716120 1176 ts_tablet_manager.cc:1950] T ef0a7b31ced24388bc9d9964a42fa203 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:15.716400 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a0b95dd9d0341d697d8709cd5e42be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.717772 1175 ts_tablet_manager.cc:1929] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.717895 1175 log.cc:1199] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/8daa66608cf94913afb94ef661b3c898
I20251029 08:52:15.718077 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6344489711f94203b5eae7cf63255c61" } new_table_name: "default.Table_9f4e595ea8564e9d902458aff13d5c78" modify_external_catalogs: true
I20251029 08:52:15.718166 1175 ts_tablet_manager.cc:1950] T 8daa66608cf94913afb94ef661b3c898 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.719094 1175 tablet_replica.cc:333] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:15.719255 1175 raft_consensus.cc:2243] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.719390 1175 raft_consensus.cc:2272] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.719892 1175 ts_tablet_manager.cc:1916] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.720913 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_49ee69a5358d49c2a46ad192a6519ae8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.721401 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49ee69a5358d49c2a46ad192a6519ae8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.721666 1176 tablet_replica.cc:333] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.721833 1176 raft_consensus.cc:2243] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.721956 1176 raft_consensus.cc:2272] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.722664 1176 ts_tablet_manager.cc:1916] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.722990 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.723142 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 8daa66608cf94913afb94ef661b3c898 (table default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]) successfully deleted
I20251029 08:52:15.723913 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ef0a7b31ced24388bc9d9964a42fa203 (table default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]) successfully deleted
I20251029 08:52:15.724459 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4be9988045844a5b90664d5b0a9238af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.726053 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4be9988045844a5b90664d5b0a9238af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.725241 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9f2f9d3d1fe34371b8a67eaf218a4154" } new_table_name: "default.Table_e67f5ce3a1ac4caab580b9e7727e6d4a" modify_external_catalogs: true
I20251029 08:52:15.730309 1176 ts_tablet_manager.cc:1929] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.730228 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c64ecb648f8c411fa562430e00d12375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.730410 1176 log.cc:1199] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/f33c0bbf104046ccb86ff313f08da22a
I20251029 08:52:15.730536 1178 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:15.730675 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c64ecb648f8c411fa562430e00d12375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.730741 1176 ts_tablet_manager.cc:1950] T f33c0bbf104046ccb86ff313f08da22a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.731655 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet f33c0bbf104046ccb86ff313f08da22a (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
I20251029 08:52:15.731925 1178 tablet_bootstrap.cc:654] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.731927 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98254c96ab3c40b6a58d2295fedd0d16" } new_table_name: "default.Table_83f92101d57a49da80bdc2552b39a1ba" modify_external_catalogs: true
I20251029 08:52:15.735920 1175 ts_tablet_manager.cc:1929] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.736057 1175 log.cc:1199] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/abc307b14aff4c5e937a5151aab1dfb9
I20251029 08:52:15.736363 1175 ts_tablet_manager.cc:1950] T abc307b14aff4c5e937a5151aab1dfb9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.737278 1175 tablet_replica.cc:333] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:15.737342 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet abc307b14aff4c5e937a5151aab1dfb9 (table default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]) successfully deleted
I20251029 08:52:15.737480 1175 raft_consensus.cc:2243] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.737640 1175 raft_consensus.cc:2272] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.738145 1175 ts_tablet_manager.cc:1916] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.740553 1176 tablet_replica.cc:333] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.740747 1176 raft_consensus.cc:2243] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.740891 1176 raft_consensus.cc:2272] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.741426 1176 ts_tablet_manager.cc:1916] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.744369 1179 ts_tablet_manager.cc:1929] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.744510 1179 log.cc:1199] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/94738d48801a4fb4a7ad1f5c4022afe1
I20251029 08:52:15.744836 1179 ts_tablet_manager.cc:1950] T 94738d48801a4fb4a7ad1f5c4022afe1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.745822 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 94738d48801a4fb4a7ad1f5c4022afe1 (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:15.747938 1176 ts_tablet_manager.cc:1929] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.748102 1176 log.cc:1199] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/5a7a673ba4fd4abcbfc3493c38dcc257
I20251029 08:52:15.748415 1176 ts_tablet_manager.cc:1950] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.749502 1177 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.749634 1177 ts_tablet_manager.cc:1403] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.001s
I20251029 08:52:15.750396 1177 raft_consensus.cc:359] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.750572 1177 raft_consensus.cc:385] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.750633 1177 raft_consensus.cc:740] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.750833 1177 consensus_queue.cc:260] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.751279 1177 ts_tablet_manager.cc:1434] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.751577 1177 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.752175 1176 tablet_replica.cc:333] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.752336 1176 raft_consensus.cc:2243] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.752435 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:15.752471 1176 raft_consensus.cc:2272] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.753188 1176 ts_tablet_manager.cc:1916] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.756682 1177 tablet_bootstrap.cc:654] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.758251 751 tablet_service.cc:1505] Processing CreateTablet for tablet 300a1a816a5347efbcae9589f1cf0c61 (DEFAULT_TABLE table=default.Table_ac5705ab49ff41f5ac459ed7fe561419 [id=adfe4ac4997f40a684db06ec3f347921]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.758713 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300a1a816a5347efbcae9589f1cf0c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.760356 888 tablet_service.cc:1505] Processing CreateTablet for tablet 300a1a816a5347efbcae9589f1cf0c61 (DEFAULT_TABLE table=default.Table_ac5705ab49ff41f5ac459ed7fe561419 [id=adfe4ac4997f40a684db06ec3f347921]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.760776 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300a1a816a5347efbcae9589f1cf0c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.761893 899 tablet_service.cc:1505] Processing CreateTablet for tablet 332e14ded2d9406b888365399710c0d2 (DEFAULT_TABLE table=default.Table_2b2df448cb2e4ac89b131af3732c83fa [id=baa8a339085b41ceaf71b437696ffba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.762288 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332e14ded2d9406b888365399710c0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.764133 1774 raft_consensus.cc:493] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.764249 1774 raft_consensus.cc:515] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.764380 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 332e14ded2d9406b888365399710c0d2 (DEFAULT_TABLE table=default.Table_2b2df448cb2e4ac89b131af3732c83fa [id=baa8a339085b41ceaf71b437696ffba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.764874 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332e14ded2d9406b888365399710c0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.764860 1774 leader_election.cc:290] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.765529 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a20f26d0f7874ccb960ffe4d27e854e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.766085 720 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.766489 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a20f26d0f7874ccb960ffe4d27e854e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.766964 718 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.767062 718 leader_election.cc:304] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.767417 1774 raft_consensus.cc:2749] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.768340 759 tablet_service.cc:1505] Processing CreateTablet for tablet 332e14ded2d9406b888365399710c0d2 (DEFAULT_TABLE table=default.Table_2b2df448cb2e4ac89b131af3732c83fa [id=baa8a339085b41ceaf71b437696ffba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.768492 1179 tablet_replica.cc:333] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:15.768671 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332e14ded2d9406b888365399710c0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.768680 1179 raft_consensus.cc:2243] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.768880 1179 raft_consensus.cc:2272] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.772922 1177 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.773036 1177 ts_tablet_manager.cc:1403] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20251029 08:52:15.773785 782 raft_consensus.cc:1275] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:15.774197 1822 consensus_queue.cc:1048] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:15.778714 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_266c99fef9414223bd1a5a73dbbf9bd5" } new_table_name: "default.Table_645b4972920f4607880280733ebe04a6" modify_external_catalogs: true
I20251029 08:52:15.774502 1177 raft_consensus.cc:359] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.778877 1177 raft_consensus.cc:385] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.778939 1177 raft_consensus.cc:740] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.779109 1177 consensus_queue.cc:260] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.779547 1177 ts_tablet_manager.cc:1434] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:15.781131 1177 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.781268 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:15.782831 1177 tablet_bootstrap.cc:654] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.783567 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_71c355ca7af84373b74ea04307f6a818"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.784245 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71c355ca7af84373b74ea04307f6a818 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.786079 761 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.786284 763 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.786331 1068 raft_consensus.cc:1275] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Refusing update from remote peer 927689e0dc5e4bc79d4a9d79a873f088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:15.786675 756 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.786830 1751 raft_consensus.cc:493] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.786983 1751 raft_consensus.cc:515] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.787664 1751 leader_election.cc:290] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.789137 1822 consensus_queue.cc:1048] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:15.791966 996 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 9165c4bf484a48b4a0b7246cb11571bf
W20251029 08:52:15.792378 998 leader_election.cc:343] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:15.792505 998 leader_election.cc:304] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.792946 1751 raft_consensus.cc:2749] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.798491 594 catalog_manager.cc:5649] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:15.799122 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:15.799683 594 catalog_manager.cc:5649] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:15.803071 1175 ts_tablet_manager.cc:1929] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.803207 1175 log.cc:1199] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/fa8884bb8e64474d936c8825d1ef3545
I20251029 08:52:15.803562 1175 ts_tablet_manager.cc:1950] T fa8884bb8e64474d936c8825d1ef3545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:15.804734 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet fa8884bb8e64474d936c8825d1ef3545 (table default.Table_d3b4e5bde9434116b192d820a909c7b1 [id=064ab59757e94a9089fed2c7b3425476]) successfully deleted
I20251029 08:52:15.806780 1822 raft_consensus.cc:493] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.806921 1822 raft_consensus.cc:515] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.807605 1822 leader_election.cc:290] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:15.808804 861 leader_election.cc:343] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c9a1d5e1f68e4f0e8b78641bbdc98123
W20251029 08:52:15.812251 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 308da528313044929b66040d6659f774 consensus state has no leader
I20251029 08:52:15.814697 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5ba2c461229a40f190ba08563f21c14f" } new_table_name: "default.Table_505dbf6dc0c64448a4ef6f67807dff75" modify_external_catalogs: true
I20251029 08:52:15.816166 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d74312a6b49e490ab01040a51e1e75e7" } modify_external_catalogs: true
I20251029 08:52:15.816355 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d74312a6b49e490ab01040a51e1e75e7" } modify_external_catalogs: true
W20251029 08:52:15.820531 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 already in progress: creating tablet
I20251029 08:52:15.822754 1179 ts_tablet_manager.cc:1916] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.833069 1177 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.833240 1177 ts_tablet_manager.cc:1403] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.014s
I20251029 08:52:15.834024 1177 raft_consensus.cc:359] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.834198 1177 raft_consensus.cc:385] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.834262 1177 raft_consensus.cc:740] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.834522 1177 consensus_queue.cc:260] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.835013 1177 ts_tablet_manager.cc:1434] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:15.835937 1177 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.836421 1176 ts_tablet_manager.cc:1929] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.836478 1178 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.836704 1178 ts_tablet_manager.cc:1403] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.122s user 0.021s sys 0.010s
I20251029 08:52:15.837544 1178 raft_consensus.cc:359] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.837759 1178 raft_consensus.cc:385] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.837845 1178 raft_consensus.cc:740] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.836539 1176 log.cc:1199] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e5fa181849474a559701fdb8c18d398b
I20251029 08:52:15.838220 1178 consensus_queue.cc:260] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.838462 1176 ts_tablet_manager.cc:1950] T e5fa181849474a559701fdb8c18d398b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.838905 1178 ts_tablet_manager.cc:1434] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.839191 1178 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.839457 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e5fa181849474a559701fdb8c18d398b (table default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf [id=ca079b1870a64ab69697d5255756c167]) successfully deleted
I20251029 08:52:15.839788 1176 tablet_replica.cc:333] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.839949 1176 raft_consensus.cc:2243] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.840107 1176 raft_consensus.cc:2272] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.840322 1822 raft_consensus.cc:493] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.840469 1822 raft_consensus.cc:515] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.840879 1176 ts_tablet_manager.cc:1916] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.841009 1822 leader_election.cc:290] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:15.841853 861 leader_election.cc:343] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ee8117e04f5f40e98ea755dc461067dd
W20251029 08:52:15.843127 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7a6ef03f18c4ee78f2f10f31d1d8058 has no consensus state
I20251029 08:52:15.843155 1178 tablet_bootstrap.cc:654] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.843679 1177 tablet_bootstrap.cc:654] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.844713 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a5e58cbcddc94f118f224404b14c6790" } modify_external_catalogs: true
I20251029 08:52:15.844913 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a5e58cbcddc94f118f224404b14c6790" } modify_external_catalogs: true
W20251029 08:52:15.845033 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 52672.9us
I20251029 08:52:15.845520 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9a1d5e1f68e4f0e8b78641bbdc98123" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.846050 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee8117e04f5f40e98ea755dc461067dd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.846385 861 leader_election.cc:343] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.846482 861 leader_election.cc:304] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.846818 1822 raft_consensus.cc:2749] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.846838 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3d5e7e9f16c04071af70d3a47624570d" } modify_external_catalogs: true
I20251029 08:52:15.847043 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3d5e7e9f16c04071af70d3a47624570d" } modify_external_catalogs: true
W20251029 08:52:15.847309 861 leader_election.cc:343] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.847600 861 leader_election.cc:304] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:15.847914 1822 raft_consensus.cc:2749] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.847482 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_29abde71edfc496a8c61702f28f25c63" } modify_external_catalogs: true
I20251029 08:52:15.848999 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_29abde71edfc496a8c61702f28f25c63" } modify_external_catalogs: true
I20251029 08:52:15.850749 1177 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:15.850881 1177 ts_tablet_manager.cc:1403] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20251029 08:52:15.852512 1177 raft_consensus.cc:359] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.852677 1177 raft_consensus.cc:385] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.852735 1177 raft_consensus.cc:740] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.853827 1177 consensus_queue.cc:260] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.854686 1177 ts_tablet_manager.cc:1434] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:15.854796 1176 ts_tablet_manager.cc:1929] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.854884 1176 log.cc:1199] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:15.855185 1176 ts_tablet_manager.cc:1950] T a20f26d0f7874ccb960ffe4d27e854e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.858726 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a20f26d0f7874ccb960ffe4d27e854e1 (table default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]) successfully deleted
I20251029 08:52:15.861188 1177 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:15.862838 1177 tablet_bootstrap.cc:654] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.866308 1176 tablet_replica.cc:333] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.866506 1176 raft_consensus.cc:2243] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.866657 1176 raft_consensus.cc:2272] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.867307 1176 ts_tablet_manager.cc:1916] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:15.875087 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce4a3503830c481797133028f5f123d9 consensus state has no leader
I20251029 08:52:15.879456 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
I20251029 08:52:15.879755 1176 ts_tablet_manager.cc:1929] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.879875 1176 log.cc:1199] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4e2773cae9814df79ee60ea8b0fe4e9d
I20251029 08:52:15.880223 1176 ts_tablet_manager.cc:1950] T 4e2773cae9814df79ee60ea8b0fe4e9d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.882095 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
I20251029 08:52:15.882356 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4e2773cae9814df79ee60ea8b0fe4e9d (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:15.882841 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43e66e8dbdf240a1b5eb4233ece857c7" } modify_external_catalogs: true
I20251029 08:52:15.884388 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43e66e8dbdf240a1b5eb4233ece857c7" } modify_external_catalogs: true
I20251029 08:52:15.886127 1928 raft_consensus.cc:493] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.887038 1928 raft_consensus.cc:515] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.888172 1934 tablet.cc:1722] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:15.890547 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d38046985bde419896eda670b4597795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.891397 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d38046985bde419896eda670b4597795 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:15.894897 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d3955ff79faf42bb82f764aae79b85d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3955ff79faf42bb82f764aae79b85d9 already in progress: creating tablet
I20251029 08:52:15.888945 751 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:15.895888 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 09968d93203140fa9201928c87702ee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09968d93203140fa9201928c87702ee0 already in progress: creating tablet
W20251029 08:52:15.899686 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59c90087c9e94665a9b3fac798304af7 consensus state has no leader
I20251029 08:52:15.903090 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 (DEFAULT_TABLE table=default.Table_54ce98414ac0405db8a6149c47af342c [id=1e68714c52894e55bfaa63fa01c4db01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.903530 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc6c8f9ee9840c08c9436ab516a62d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.913136 888 tablet_service.cc:1505] Processing CreateTablet for tablet a013257089774ba8b1f535e5f94ac1bb (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.913599 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013257089774ba8b1f535e5f94ac1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.901566 1933 tablet.cc:1722] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251029 08:52:15.922158 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 787d6d7f1870450fa21cb9cba34e72fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 787d6d7f1870450fa21cb9cba34e72fe already in progress: creating tablet
W20251029 08:52:15.922878 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 already in progress: creating tablet
W20251029 08:52:15.923449 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f4c1b1d8cf7d464ebad3afca1e31258f because the tablet was not found. No further retry: Not found: Tablet not found: f4c1b1d8cf7d464ebad3afca1e31258f
I20251029 08:52:15.923736 1176 tablet_replica.cc:333] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.923972 1176 raft_consensus.cc:2243] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.924206 1176 raft_consensus.cc:2272] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.924963 1178 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.925130 1178 ts_tablet_manager.cc:1403] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.002s
I20251029 08:52:15.925793 1178 raft_consensus.cc:359] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.926012 1178 raft_consensus.cc:385] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.926096 1178 raft_consensus.cc:740] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.926352 1178 consensus_queue.cc:260] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:15.927091 1178 ts_tablet_manager.cc:1434] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:15.929816 1178 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:15.931241 1176 ts_tablet_manager.cc:1916] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.931427 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 (DEFAULT_TABLE table=default.Table_54ce98414ac0405db8a6149c47af342c [id=1e68714c52894e55bfaa63fa01c4db01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.931912 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc6c8f9ee9840c08c9436ab516a62d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.933964 1928 leader_election.cc:290] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.934549 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:15.935552 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9ed0a270b794e469d2fdbd0ea1d96f8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:15.936074 998 leader_election.cc:304] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.936517 1928 raft_consensus.cc:2749] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.940151 762 tablet_service.cc:1505] Processing CreateTablet for tablet a013257089774ba8b1f535e5f94ac1bb (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.940323 1178 tablet_bootstrap.cc:654] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:15.940647 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013257089774ba8b1f535e5f94ac1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.942373 1176 ts_tablet_manager.cc:1929] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.942469 1176 log.cc:1199] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2c706343d44a4807b1ea60c1b1e1ea49
I20251029 08:52:15.942775 1176 ts_tablet_manager.cc:1950] T 2c706343d44a4807b1ea60c1b1e1ea49 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:15.943671 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2c706343d44a4807b1ea60c1b1e1ea49 (table default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]) successfully deleted
I20251029 08:52:15.944238 1935 tablet.cc:1722] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:15.948424 1774 raft_consensus.cc:493] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.948549 1774 raft_consensus.cc:515] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.949123 1774 leader_election.cc:290] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.949676 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c572c68b32d443b2a019e1f4bf2d82fe" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.950256 718 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.950974 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c572c68b32d443b2a019e1f4bf2d82fe" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.951505 1774 raft_consensus.cc:493] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.951646 1774 raft_consensus.cc:515] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.952188 1774 leader_election.cc:290] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:15.952682 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:15.953214 718 leader_election.cc:343] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:15.953593 720 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.953687 720 leader_election.cc:304] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.954285 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.954768 1176 tablet_replica.cc:333] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:15.954768 720 leader_election.cc:343] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.954875 720 leader_election.cc:304] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.954990 1176 raft_consensus.cc:2243] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.955222 1176 raft_consensus.cc:2272] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.955302 593 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
I20251029 08:52:15.955575 1774 raft_consensus.cc:2749] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.955718 1774 raft_consensus.cc:2749] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.955992 1176 ts_tablet_manager.cc:1916] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:15.956944 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]). Expected version 3 got 0
W20251029 08:52:15.966578 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81c2c690e3364f3abb61bc01bbb2980e has no consensus state
I20251029 08:52:15.967299 1779 raft_consensus.cc:493] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.967432 1779 raft_consensus.cc:515] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.968052 1779 leader_election.cc:290] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.968145 1176 ts_tablet_manager.cc:1929] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.968603 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8626877e3cfc4cb48f6caa92638b4917" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:15.969161 1779 raft_consensus.cc:493] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.969274 1779 raft_consensus.cc:515] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:15.969326 861 leader_election.cc:343] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.969848 1779 leader_election.cc:290] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.971900 1928 raft_consensus.cc:493] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.972033 1928 raft_consensus.cc:515] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.972522 1928 leader_election.cc:290] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:15.973459 996 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 63b08840d0fa40428c61a05b277ad750
I20251029 08:52:15.972434 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6541220eef6e47c1861484bbf1a5f44c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.974179 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc96ef1a94844a32923087c44a5bcfc8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.974244 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6541220eef6e47c1861484bbf1a5f44c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:15.974653 861 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:15.974687 998 leader_election.cc:343] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 63b08840d0fa40428c61a05b277ad750
I20251029 08:52:15.974805 998 leader_election.cc:304] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:15.975347 1928 raft_consensus.cc:2749] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:15.975570 1176 log.cc:1199] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e4f80728a7a04583bada485707beb806
I20251029 08:52:15.975951 1176 ts_tablet_manager.cc:1950] T e4f80728a7a04583bada485707beb806 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:15.976312 582 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_83f92101d57a49da80bdc2552b39a1ba [id=e1da82f219d140ad8ee2c8d2786bd40e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ce4a3503830c481797133028f5f123d9 consensus state has no leader
I20251029 08:52:15.977474 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a9fe3e9a507441b8ba5953c0120289c3
I20251029 08:52:15.978066 1178 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:15.978183 1178 ts_tablet_manager.cc:1403] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.001s
I20251029 08:52:15.978277 902 tablet_service.cc:1552] Processing DeleteTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.978657 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e4f80728a7a04583bada485707beb806 (table default.Table_18a7eae91e5b4ab595c36f630248860f [id=99ae7a11d37346e18a01a330d7fa42b1]) successfully deleted
W20251029 08:52:15.978853 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a9fe3e9a507441b8ba5953c0120289c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9fe3e9a507441b8ba5953c0120289c3 already in progress: creating tablet
I20251029 08:52:15.978888 1178 raft_consensus.cc:359] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.979046 1178 raft_consensus.cc:385] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:15.979099 1178 raft_consensus.cc:740] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:15.979171 1176 tablet_replica.cc:333] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:15.979323 1176 raft_consensus.cc:2243] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:15.979290 1178 consensus_queue.cc:260] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.979435 1176 raft_consensus.cc:2272] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:15.980437 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:15.980782 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:15.981617 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:15.981612 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.982116 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:15.982292 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9429f6abb50c4f66b2e1b565f303962d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9429f6abb50c4f66b2e1b565f303962d already in progress: creating tablet
I20251029 08:52:15.982729 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f713c7b9b580405c87b7fdc7df46acf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:15.983284 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f713c7b9b580405c87b7fdc7df46acf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.983354 1179 ts_tablet_manager.cc:1929] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:15.983458 1179 log.cc:1199] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/23b534aaa51b4fad9dc3fc915de947f1
I20251029 08:52:15.983767 1179 ts_tablet_manager.cc:1950] T 23b534aaa51b4fad9dc3fc915de947f1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:15.984740 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4d371c09f6534a4db843f2ee86ef54df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:15.984984 1178 ts_tablet_manager.cc:1434] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20251029 08:52:15.985505 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d371c09f6534a4db843f2ee86ef54df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:15.987334 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:15.987859 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 (DEFAULT_TABLE table=default.Table_54ce98414ac0405db8a6149c47af342c [id=1e68714c52894e55bfaa63fa01c4db01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:15.988312 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc6c8f9ee9840c08c9436ab516a62d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:15.988379 1176 ts_tablet_manager.cc:1916] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:15.991400 861 consensus_peers.cc:597] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:15.991649 1779 raft_consensus.cc:493] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:15.991781 1779 raft_consensus.cc:515] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:15.992455 1779 leader_election.cc:290] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:15.993233 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757e34137e884d5ca5cd88fdd03244d8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:15.993698 861 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:15.998775 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0ab56045bd814d59831a98618c759af4
I20251029 08:52:15.999543 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:16.000233 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0ab56045bd814d59831a98618c759af4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab56045bd814d59831a98618c759af4 already in progress: creating tablet
I20251029 08:52:16.000586 1176 ts_tablet_manager.cc:1929] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.000700 1176 log.cc:1199] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:16.000996 1176 ts_tablet_manager.cc:1950] T c572c68b32d443b2a019e1f4bf2d82fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:16.001073 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.001513 1178 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.001698 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:16.002419 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.002897 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.003482 1178 tablet_bootstrap.cc:654] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:16.003551 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7be4bcf9e3324bd884e946fae20648fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be4bcf9e3324bd884e946fae20648fd already in progress: creating tablet
I20251029 08:52:16.003667 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.003934 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.010416 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d9b9417a07e84ab38c3498a14cbbb03d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.011174 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9b9417a07e84ab38c3498a14cbbb03d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.015334 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 23b534aaa51b4fad9dc3fc915de947f1 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
I20251029 08:52:16.015695 1928 raft_consensus.cc:493] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.015847 1928 raft_consensus.cc:515] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.016623 1928 leader_election.cc:290] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.017007 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c572c68b32d443b2a019e1f4bf2d82fe (table default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]) successfully deleted
I20251029 08:52:16.017323 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3cac145320cf48b4aaf4adb0b686ba8c
I20251029 08:52:16.018038 1176 tablet_replica.cc:333] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.018224 1176 raft_consensus.cc:2243] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.018380 1176 raft_consensus.cc:2272] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.018512 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:16.019171 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3cac145320cf48b4aaf4adb0b686ba8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cac145320cf48b4aaf4adb0b686ba8c already in progress: creating tablet
W20251029 08:52:16.019328 996 leader_election.cc:343] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:16.019716 1176 ts_tablet_manager.cc:1916] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:16.019987 998 leader_election.cc:343] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:16.020114 998 leader_election.cc:304] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.020512 1928 raft_consensus.cc:2749] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.022428 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.022670 593 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7] alter complete (version 3)
I20251029 08:52:16.024472 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.024802 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.024955 1178 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.025075 1178 ts_tablet_manager.cc:1403] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20251029 08:52:16.024878 1179 tablet_replica.cc:333] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.025249 1179 raft_consensus.cc:2243] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.025416 1179 raft_consensus.cc:2272] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.025795 1178 raft_consensus.cc:359] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:16.025929 1179 ts_tablet_manager.cc:1916] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.025950 1178 raft_consensus.cc:385] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.025998 1178 raft_consensus.cc:740] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.026227 1178 consensus_queue.cc:260] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:16.027503 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_49ee69a5358d49c2a46ad192a6519ae8" } new_table_name: "default.Table_a1a3d90a54344467a739bc9d4d5a1b91" modify_external_catalogs: true
W20251029 08:52:16.028086 1183 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.079s sys 0.167s
W20251029 08:52:16.028170 1183 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.607s user 0.079s sys 0.167s
I20251029 08:52:16.028975 1183 tablet_bootstrap.cc:492] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.029127 1183 ts_tablet_manager.cc:1403] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.611s user 0.081s sys 0.167s
I20251029 08:52:16.029815 1178 ts_tablet_manager.cc:1434] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:16.029906 1183 raft_consensus.cc:359] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.030082 1183 raft_consensus.cc:385] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.030162 1183 raft_consensus.cc:740] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.030051 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:16.030484 1183 consensus_queue.cc:260] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.031105 1183 ts_tablet_manager.cc:1434] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.031564 1183 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.031575 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e5472287b7984f8f8a0fbd29cee00c13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.032306 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5472287b7984f8f8a0fbd29cee00c13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.033025 1178 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.033210 1183 tablet_bootstrap.cc:654] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.035342 1178 tablet_bootstrap.cc:654] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.036818 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:16.039080 1179 ts_tablet_manager.cc:1929] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.039191 1179 log.cc:1199] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/63b08840d0fa40428c61a05b277ad750
I20251029 08:52:16.039505 1179 ts_tablet_manager.cc:1950] T 63b08840d0fa40428c61a05b277ad750 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.042225 1183 tablet_bootstrap.cc:492] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.042352 1183 ts_tablet_manager.cc:1403] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251029 08:52:16.043063 1176 ts_tablet_manager.cc:1929] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.043181 1176 log.cc:1199] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c7024e6eb4124109ad1575d216739fe2
I20251029 08:52:16.043130 1183 raft_consensus.cc:359] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.043329 1183 raft_consensus.cc:385] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.043388 1183 raft_consensus.cc:740] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.043550 1176 ts_tablet_manager.cc:1950] T c7024e6eb4124109ad1575d216739fe2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:16.044523 1779 raft_consensus.cc:493] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.044708 1779 raft_consensus.cc:515] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.045167 1183 consensus_queue.cc:260] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.045392 1779 leader_election.cc:290] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.045665 1183 ts_tablet_manager.cc:1434] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:16.045912 1183 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.046941 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.047652 1183 tablet_bootstrap.cc:654] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:16.048045 861 leader_election.cc:343] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.052881 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c7024e6eb4124109ad1575d216739fe2 (table default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]) successfully deleted
I20251029 08:52:16.053453 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_61da851751cb48c99d7c6dded2f21a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.053769 1178 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.053887 1178 ts_tablet_manager.cc:1403] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
W20251029 08:52:16.054435 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61da851751cb48c99d7c6dded2f21a43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.054577 1178 raft_consensus.cc:359] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.054734 1178 raft_consensus.cc:385] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.054785 1178 raft_consensus.cc:740] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.054987 1178 consensus_queue.cc:260] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.056977 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 63b08840d0fa40428c61a05b277ad750 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:16.057636 1179 tablet_replica.cc:333] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.057857 1178 ts_tablet_manager.cc:1434] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:16.058775 1179 raft_consensus.cc:2243] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.059159 1179 raft_consensus.cc:2272] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.059878 1179 ts_tablet_manager.cc:1916] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.061367 1183 tablet_bootstrap.cc:492] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.064585 1183 ts_tablet_manager.cc:1403] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251029 08:52:16.064934 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_079d27f45f6a49e093d99a073678a387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.065554 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_079d27f45f6a49e093d99a073678a387 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.066177 1178 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.066656 1183 raft_consensus.cc:359] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.067397 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.068147 1178 tablet_bootstrap.cc:654] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.068993 1176 tablet_replica.cc:333] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.069173 1176 raft_consensus.cc:2243] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.067837 1183 raft_consensus.cc:385] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.069320 1176 raft_consensus.cc:2272] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.069465 1183 raft_consensus.cc:740] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.070035 1176 ts_tablet_manager.cc:1916] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.075644 1183 consensus_queue.cc:260] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.078866 1178 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.079005 1178 ts_tablet_manager.cc:1403] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20251029 08:52:16.079198 1176 ts_tablet_manager.cc:1929] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.079290 1176 log.cc:1199] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:16.079586 1176 ts_tablet_manager.cc:1950] T e626994e96d54c0d913fdedc85fd1155 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:16.079926 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 125172us
I20251029 08:52:16.079978 1183 ts_tablet_manager.cc:1434] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251029 08:52:16.080183 1178 raft_consensus.cc:359] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.080343 1178 raft_consensus.cc:385] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.080392 1178 raft_consensus.cc:740] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.080543 1178 consensus_queue.cc:260] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.080955 1178 ts_tablet_manager.cc:1434] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:16.082468 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_69a998c83a8c499d86a9931423a1c53f" } new_table_name: "default.Table_e0afe3a68cdb4a65a182f56e3da6a709" modify_external_catalogs: true
I20251029 08:52:16.084285 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5ea68420364f46228f00542b348dc801" } new_table_name: "default.Table_bf9b59ca50fd463eb0768ec552e1244a" modify_external_catalogs: true
I20251029 08:52:16.085714 1178 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.088567 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:16.088737 780 raft_consensus.cc:2468] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:16.089155 861 leader_election.cc:343] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 8626877e3cfc4cb48f6caa92638b4917
I20251029 08:52:16.089282 861 leader_election.cc:304] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.089656 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 (DEFAULT_TABLE table=default.Table_98e2b0275a064384b92201c42be8a086 [id=9b2d95c857074a7fb3a2d5a5c6ebfb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.089787 1779 raft_consensus.cc:2749] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:16.090194 861 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:16.090299 861 leader_election.cc:304] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.090692 1779 raft_consensus.cc:2749] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.091444 1179 ts_tablet_manager.cc:1929] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.091439 1822 consensus_queue.cc:1048] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:16.091566 1179 log.cc:1199] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9165c4bf484a48b4a0b7246cb11571bf
I20251029 08:52:16.091869 1179 ts_tablet_manager.cc:1950] T 9165c4bf484a48b4a0b7246cb11571bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.091892 902 tablet_service.cc:1505] Processing CreateTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 (DEFAULT_TABLE table=default.Table_98e2b0275a064384b92201c42be8a086 [id=9b2d95c857074a7fb3a2d5a5c6ebfb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.092147 1178 tablet_bootstrap.cc:654] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.092342 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adba75cc7d74b1497de9af7ad7ccdd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:16.092698 861 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:16.092793 861 leader_election.cc:304] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.093118 1779 raft_consensus.cc:2749] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.093461 861 leader_election.cc:304] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:16.093868 1779 raft_consensus.cc:2804] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:16.093937 1779 raft_consensus.cc:493] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:16.094010 1779 raft_consensus.cc:3060] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.096778 1779 raft_consensus.cc:515] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.097404 1779 leader_election.cc:290] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.104439 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 (DEFAULT_TABLE table=default.Table_98e2b0275a064384b92201c42be8a086 [id=9b2d95c857074a7fb3a2d5a5c6ebfb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.105046 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adba75cc7d74b1497de9af7ad7ccdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.105208 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:16.105371 782 raft_consensus.cc:3060] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.106557 1779 consensus_queue.cc:1048] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:16.109352 1051 tablet_service.cc:1505] Processing CreateTablet for tablet a013257089774ba8b1f535e5f94ac1bb (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.118149 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
I20251029 08:52:16.129860 1779 raft_consensus.cc:493] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.130115 1779 raft_consensus.cc:515] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.130846 1779 leader_election.cc:290] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:16.135005 861 leader_election.cc:343] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3295bb0b47714080a5c3f34049acd487
I20251029 08:52:16.151014 1048 tablet_service.cc:1505] Processing CreateTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f (DEFAULT_TABLE table=default.Table_40ff04cf3a70422293d9648f7364d8a0 [id=29354cd8a4914187b162ecab327ab01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.153661 759 tablet_service.cc:1505] Processing CreateTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f (DEFAULT_TABLE table=default.Table_40ff04cf3a70422293d9648f7364d8a0 [id=29354cd8a4914187b162ecab327ab01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.154210 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c1b1d8cf7d464ebad3afca1e31258f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.162459 1033 tablet_service.cc:1505] Processing CreateTablet for tablet eb80603843ab428d8329b5c387a28a7b (DEFAULT_TABLE table=default.Table_bb70d01442d841199629d07f2ee8ed72 [id=0f0b822dd9084e5e9407e7605bd382b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.165272 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:16.165565 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3295bb0b47714080a5c3f34049acd487" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.166126 756 tablet_service.cc:1505] Processing CreateTablet for tablet eb80603843ab428d8329b5c387a28a7b (DEFAULT_TABLE table=default.Table_bb70d01442d841199629d07f2ee8ed72 [id=0f0b822dd9084e5e9407e7605bd382b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.166499 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb80603843ab428d8329b5c387a28a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.171377 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adba75cc7d74b1497de9af7ad7ccdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.171952 1179 tablet_replica.cc:333] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.172158 1179 raft_consensus.cc:2243] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.172340 1179 raft_consensus.cc:2272] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.172923 1179 ts_tablet_manager.cc:1916] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:16.173373 861 leader_election.cc:343] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:16.173612 861 leader_election.cc:343] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.173709 861 leader_election.cc:304] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.174245 1948 consensus_queue.cc:1048] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:16.174522 1779 raft_consensus.cc:2749] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.175578 1071 raft_consensus.cc:3060] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.176226 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb80603843ab428d8329b5c387a28a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.176689 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013257089774ba8b1f535e5f94ac1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.178748 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c1b1d8cf7d464ebad3afca1e31258f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.184283 1183 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.185911 1183 tablet_bootstrap.cc:654] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.186995 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9165c4bf484a48b4a0b7246cb11571bf (table default.Table_742704efe2d14569b8a72c8e7d09c736 [id=63a5ee7ef2b34d48a993e21fde8e42f0]) successfully deleted
I20251029 08:52:16.188004 902 tablet_service.cc:1505] Processing CreateTablet for tablet f4c1b1d8cf7d464ebad3afca1e31258f (DEFAULT_TABLE table=default.Table_40ff04cf3a70422293d9648f7364d8a0 [id=29354cd8a4914187b162ecab327ab01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.188797 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c1b1d8cf7d464ebad3afca1e31258f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.193707 782 raft_consensus.cc:2468] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:16.193979 1179 ts_tablet_manager.cc:1929] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.194092 1179 log.cc:1199] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/28face87091b49d2971217618c95d2f5
I20251029 08:52:16.194394 1179 ts_tablet_manager.cc:1950] T 28face87091b49d2971217618c95d2f5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.195441 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 28face87091b49d2971217618c95d2f5 (table default.Table_6899fb76bad448cd9d1ca4299c5f7562 [id=b95070146a0d49be8c07791ebf078593]) successfully deleted
I20251029 08:52:16.199786 1179 tablet_replica.cc:333] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.200289 1179 raft_consensus.cc:2243] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.200750 1179 raft_consensus.cc:2272] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.201535 1179 ts_tablet_manager.cc:1916] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.203063 861 leader_election.cc:304] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:16.203615 1779 raft_consensus.cc:2804] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:16.203809 1779 raft_consensus.cc:697] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:16.204625 1779 consensus_queue.cc:237] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.209573 1183 tablet_bootstrap.cc:492] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.209720 1183 ts_tablet_manager.cc:1403] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.002s
I20251029 08:52:16.210458 1183 raft_consensus.cc:359] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.210628 1183 raft_consensus.cc:385] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.210686 1183 raft_consensus.cc:740] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.210923 1183 consensus_queue.cc:260] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.211506 1183 ts_tablet_manager.cc:1434] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.211791 1183 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.214174 1176 tablet_replica.cc:333] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.214318 1183 tablet_bootstrap.cc:654] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.214572 1176 raft_consensus.cc:2243] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.214812 1176 raft_consensus.cc:2272] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.215720 1176 ts_tablet_manager.cc:1916] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.216472 1179 ts_tablet_manager.cc:1929] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.216586 1179 log.cc:1199] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/be706771a95e45e5a20a15614eadfe1a
I20251029 08:52:16.216872 1179 ts_tablet_manager.cc:1950] T be706771a95e45e5a20a15614eadfe1a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.217363 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e626994e96d54c0d913fdedc85fd1155 (table default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]) successfully deleted
W20251029 08:52:16.217643 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9429f6abb50c4f66b2e1b565f303962d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9429f6abb50c4f66b2e1b565f303962d already in progress: creating tablet
W20251029 08:52:16.217904 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a9fe3e9a507441b8ba5953c0120289c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9fe3e9a507441b8ba5953c0120289c3 already in progress: creating tablet
W20251029 08:52:16.218097 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7be4bcf9e3324bd884e946fae20648fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be4bcf9e3324bd884e946fae20648fd already in progress: creating tablet
W20251029 08:52:16.218353 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0ab56045bd814d59831a98618c759af4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab56045bd814d59831a98618c759af4 already in progress: creating tablet
I20251029 08:52:16.218560 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet be706771a95e45e5a20a15614eadfe1a (table default.Table_a69aba63f41c4596a1017f38f6ed6532 [id=e20293dc36d54acf9ee6f9bf7ac55834]) successfully deleted
I20251029 08:52:16.218743 1183 tablet_bootstrap.cc:492] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.218847 1183 ts_tablet_manager.cc:1403] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251029 08:52:16.219535 1183 raft_consensus.cc:359] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.219689 1183 raft_consensus.cc:385] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.219739 1183 raft_consensus.cc:740] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.219960 1183 consensus_queue.cc:260] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.220583 1179 tablet_replica.cc:333] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.220760 1179 raft_consensus.cc:2243] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.220901 1179 raft_consensus.cc:2272] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.221426 1179 ts_tablet_manager.cc:1916] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.224146 1177 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.224314 1177 ts_tablet_manager.cc:1403] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.363s user 0.057s sys 0.078s
I20251029 08:52:16.225068 1177 raft_consensus.cc:359] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.225263 1177 raft_consensus.cc:385] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.225369 1177 raft_consensus.cc:740] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.225610 1177 consensus_queue.cc:260] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.226150 1177 ts_tablet_manager.cc:1434] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.226450 1177 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.227126 1822 raft_consensus.cc:493] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.227371 1822 raft_consensus.cc:515] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.228220 1177 tablet_bootstrap.cc:654] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.228446 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.228627 1070 raft_consensus.cc:2468] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:16.229048 861 leader_election.cc:304] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:16.229780 1183 ts_tablet_manager.cc:1434] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:16.230002 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:16.230079 1183 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.230197 782 raft_consensus.cc:2468] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:16.230674 1946 raft_consensus.cc:2804] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:16.230897 1946 raft_consensus.cc:493] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:16.231040 1822 leader_election.cc:290] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.231124 1946 raft_consensus.cc:3060] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.232095 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7aa2b72bba9d4f6d80cc2a4a09825a94" } new_table_name: "default.Table_2c072545dc7446eabcdbe80f548c5df7" modify_external_catalogs: true
I20251029 08:52:16.234045 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_22ad2d235d9c4ccdb70814d03790c887"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.234179 1946 raft_consensus.cc:515] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:16.235034 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22ad2d235d9c4ccdb70814d03790c887 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.236330 1183 tablet_bootstrap.cc:654] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.236900 1178 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.237039 1178 ts_tablet_manager.cc:1403] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.151s user 0.030s sys 0.024s
I20251029 08:52:16.238215 1946 leader_election.cc:290] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.238678 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:16.238858 1070 raft_consensus.cc:3060] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.242283 1070 raft_consensus.cc:2468] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:16.242398 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cea6d85081f246788ba29bb4a9c9a4e8 has no consensus state
I20251029 08:52:16.242763 861 leader_election.cc:304] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:16.243041 1946 raft_consensus.cc:2804] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:16.243304 1946 raft_consensus.cc:697] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:16.243402 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca5bfc2ee3c6491eb6857f6b9855daf9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:16.243564 782 raft_consensus.cc:3060] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.243607 1946 consensus_queue.cc:237] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.245791 1176 ts_tablet_manager.cc:1929] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.245890 1176 log.cc:1199] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:16.246204 1176 ts_tablet_manager.cc:1950] T 2723cdd513ee4e8ca6773b38f785b09e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:16.246176 782 raft_consensus.cc:2468] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:16.237893 1178 raft_consensus.cc:359] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.246862 1178 raft_consensus.cc:385] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.246925 1178 raft_consensus.cc:740] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.247057 1183 tablet_bootstrap.cc:492] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.247190 1183 ts_tablet_manager.cc:1403] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251029 08:52:16.247107 1178 consensus_queue.cc:260] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.247224 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2723cdd513ee4e8ca6773b38f785b09e (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:16.247875 1183 raft_consensus.cc:359] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.248047 1183 raft_consensus.cc:385] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.248100 1183 raft_consensus.cc:740] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.249454 1177 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.249657 1070 raft_consensus.cc:3060] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.249672 1177 ts_tablet_manager.cc:1403] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20251029 08:52:16.250404 1177 raft_consensus.cc:359] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.250610 1177 raft_consensus.cc:385] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.250679 1177 raft_consensus.cc:740] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.250927 1177 consensus_queue.cc:260] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.251549 1177 ts_tablet_manager.cc:1434] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.251912 1177 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.254956 1177 tablet_bootstrap.cc:654] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.255120 1178 ts_tablet_manager.cc:1434] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251029 08:52:16.256700 1178 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.258253 1179 ts_tablet_manager.cc:1929] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.258311 1178 tablet_bootstrap.cc:654] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.258380 1179 log.cc:1199] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7cbce5d595364b4bad36d59fb532c237
I20251029 08:52:16.258661 1179 ts_tablet_manager.cc:1950] T 7cbce5d595364b4bad36d59fb532c237 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.258785 595 master_service.cc:954] ReplaceTablet: received request to replace tablet e5fa181849474a559701fdb8c18d398b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.259621 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7cbce5d595364b4bad36d59fb532c237 (table default.Table_d65e71e019114f57877994e1b43540ef [id=eb3e8cece7f44dd6855a2e2e7c7be39e]) successfully deleted
I20251029 08:52:16.259680 1183 consensus_queue.cc:260] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.260504 1183 ts_tablet_manager.cc:1434] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251029 08:52:16.260469 1935 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:16.260789 1183 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.262833 1183 tablet_bootstrap.cc:654] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.267204 1179 tablet_replica.cc:333] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:16.267204 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3cac145320cf48b4aaf4adb0b686ba8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cac145320cf48b4aaf4adb0b686ba8c already in progress: creating tablet
I20251029 08:52:16.267433 1179 raft_consensus.cc:2243] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.267609 1179 raft_consensus.cc:2272] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.268208 1176 tablet_replica.cc:333] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.268214 1179 ts_tablet_manager.cc:1916] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.271392 888 tablet_service.cc:1505] Processing CreateTablet for tablet eb80603843ab428d8329b5c387a28a7b (DEFAULT_TABLE table=default.Table_bb70d01442d841199629d07f2ee8ed72 [id=0f0b822dd9084e5e9407e7605bd382b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.271960 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb80603843ab428d8329b5c387a28a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.274377 899 tablet_service.cc:1505] Processing CreateTablet for tablet d37fdd5241d34200b5d2f7f57a2f9607 (DEFAULT_TABLE table=default.Table_d089089a84af48068f5a61c67b1ea446 [id=9d22aea3e7094affa6a6a0853b9b777a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.274806 895 tablet_service.cc:1505] Processing CreateTablet for tablet a4ae5978b71443a48a88bc69128b187d (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.274932 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37fdd5241d34200b5d2f7f57a2f9607. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.275262 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae5978b71443a48a88bc69128b187d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.277503 1048 tablet_service.cc:1505] Processing CreateTablet for tablet d37fdd5241d34200b5d2f7f57a2f9607 (DEFAULT_TABLE table=default.Table_d089089a84af48068f5a61c67b1ea446 [id=9d22aea3e7094affa6a6a0853b9b777a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.277945 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37fdd5241d34200b5d2f7f57a2f9607. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.277912 1033 tablet_service.cc:1505] Processing CreateTablet for tablet a4ae5978b71443a48a88bc69128b187d (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.278301 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae5978b71443a48a88bc69128b187d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.300194 1179 ts_tablet_manager.cc:1929] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.277017 762 tablet_service.cc:1505] Processing CreateTablet for tablet a4ae5978b71443a48a88bc69128b187d (DEFAULT_TABLE table=default.Table_7f838f98d4a34099a018852bfa404b97 [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.304926 1176 raft_consensus.cc:2243] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.305088 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae5978b71443a48a88bc69128b187d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.305114 1176 raft_consensus.cc:2272] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.306089 1176 ts_tablet_manager.cc:1916] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.306855 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_abb3201c55f04e0083c3515dd7f64bee" } new_table_name: "default.Table_5471f52640fd473e90327f739bd8223d" modify_external_catalogs: true
I20251029 08:52:16.277522 759 tablet_service.cc:1505] Processing CreateTablet for tablet d37fdd5241d34200b5d2f7f57a2f9607 (DEFAULT_TABLE table=default.Table_d089089a84af48068f5a61c67b1ea446 [id=9d22aea3e7094affa6a6a0853b9b777a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.308473 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37fdd5241d34200b5d2f7f57a2f9607. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.308928 1179 log.cc:1199] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/add0605edf5c4511ae4c8d9cb7f0babb
I20251029 08:52:16.309275 1179 ts_tablet_manager.cc:1950] T add0605edf5c4511ae4c8d9cb7f0babb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.313854 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet add0605edf5c4511ae4c8d9cb7f0babb (table default.Table_3eda8a3eed924142880c9b465eb759b9 [id=edda8e15b59b472886bb1eba8fec8739]) successfully deleted
W20251029 08:52:16.314680 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a9fe3e9a507441b8ba5953c0120289c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9fe3e9a507441b8ba5953c0120289c3 already in progress: creating tablet
W20251029 08:52:16.316079 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9429f6abb50c4f66b2e1b565f303962d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9429f6abb50c4f66b2e1b565f303962d already in progress: creating tablet
W20251029 08:52:16.317246 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0ab56045bd814d59831a98618c759af4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab56045bd814d59831a98618c759af4 already in progress: creating tablet
W20251029 08:52:16.318158 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7be4bcf9e3324bd884e946fae20648fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be4bcf9e3324bd884e946fae20648fd already in progress: creating tablet
W20251029 08:52:16.318945 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3cac145320cf48b4aaf4adb0b686ba8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cac145320cf48b4aaf4adb0b686ba8c already in progress: creating tablet
I20251029 08:52:16.324635 1178 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.324790 1178 ts_tablet_manager.cc:1403] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.009s
I20251029 08:52:16.325618 1178 raft_consensus.cc:359] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.325796 1178 raft_consensus.cc:385] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.325843 1178 raft_consensus.cc:740] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.326062 1178 consensus_queue.cc:260] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.327941 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8ffc14da79c544d394a4d7fe49b465f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.328217 1176 ts_tablet_manager.cc:1929] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.328334 1176 log.cc:1199] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/5385fef404a4463595074f0c8ff822d7
I20251029 08:52:16.328943 1176 ts_tablet_manager.cc:1950] T 5385fef404a4463595074f0c8ff822d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:16.329195 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ffc14da79c544d394a4d7fe49b465f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.329977 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 5385fef404a4463595074f0c8ff822d7 (table default.Table_58fa7040e0e2467588dd8b5a0a2eba4b [id=2cfb2b85354d43e7a203d35a4448346d]) successfully deleted
I20251029 08:52:16.330950 1178 ts_tablet_manager.cc:1434] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:16.334221 1178 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.340857 1178 tablet_bootstrap.cc:654] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.344996 759 tablet_service.cc:1505] Processing CreateTablet for tablet cf7604670833495c9dd61258a3301bb4 (DEFAULT_TABLE table=default.Table_11375103cdf341e8a98dca7cca956197 [id=8f5ca9bcad47421ab2e88280da8126f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.345541 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7604670833495c9dd61258a3301bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.352569 1178 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.352736 1178 ts_tablet_manager.cc:1403] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20251029 08:52:16.353549 1178 raft_consensus.cc:359] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.353729 1178 raft_consensus.cc:385] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.353780 1178 raft_consensus.cc:740] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.356789 1178 consensus_queue.cc:260] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.361464 1178 ts_tablet_manager.cc:1434] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 08:52:16.361954 1051 tablet_service.cc:1505] Processing CreateTablet for tablet cf7604670833495c9dd61258a3301bb4 (DEFAULT_TABLE table=default.Table_11375103cdf341e8a98dca7cca956197 [id=8f5ca9bcad47421ab2e88280da8126f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.362625 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7604670833495c9dd61258a3301bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.368996 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6f454ac2adfa4a4da06fa1180ca26c58" } modify_external_catalogs: true
I20251029 08:52:16.369254 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6f454ac2adfa4a4da06fa1180ca26c58" } modify_external_catalogs: true
I20251029 08:52:16.369395 1935 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:16.377230 1178 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.378894 1178 tablet_bootstrap.cc:654] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.395660 1178 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.395814 1178 ts_tablet_manager.cc:1403] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20251029 08:52:16.397161 1178 raft_consensus.cc:359] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.397336 1178 raft_consensus.cc:385] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.397380 1178 raft_consensus.cc:740] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.397615 1178 consensus_queue.cc:260] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.402807 1178 ts_tablet_manager.cc:1434] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251029 08:52:16.406067 1178 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.407727 1178 tablet_bootstrap.cc:654] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.417718 896 tablet_service.cc:1505] Processing CreateTablet for tablet cf7604670833495c9dd61258a3301bb4 (DEFAULT_TABLE table=default.Table_11375103cdf341e8a98dca7cca956197 [id=8f5ca9bcad47421ab2e88280da8126f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.418231 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7604670833495c9dd61258a3301bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.420444 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 5da5efbac1714fbbaa6bea49e473517b (DEFAULT_TABLE table=default.Table_55b3b56ad90e4db1b7b8282560023ecc [id=b24e2570c811491daec8458154654e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.420814 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d97ee1ab84f9ba23946941938159c (DEFAULT_TABLE table=default.Table_4f3ecd8febb64821a555373de62d6b6e [id=201a9d9dafaf4ecd9c28ff56c44f57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.420846 751 tablet_service.cc:1505] Processing CreateTablet for tablet 5da5efbac1714fbbaa6bea49e473517b (DEFAULT_TABLE table=default.Table_55b3b56ad90e4db1b7b8282560023ecc [id=b24e2570c811491daec8458154654e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.420936 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da5efbac1714fbbaa6bea49e473517b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.421193 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d97ee1ab84f9ba23946941938159c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.421195 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da5efbac1714fbbaa6bea49e473517b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.421597 888 tablet_service.cc:1505] Processing CreateTablet for tablet 5da5efbac1714fbbaa6bea49e473517b (DEFAULT_TABLE table=default.Table_55b3b56ad90e4db1b7b8282560023ecc [id=b24e2570c811491daec8458154654e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.421948 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da5efbac1714fbbaa6bea49e473517b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.430351 1178 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.430531 1178 ts_tablet_manager.cc:1403] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20251029 08:52:16.431280 1946 raft_consensus.cc:493] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.431444 1946 raft_consensus.cc:515] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.431563 1178 raft_consensus.cc:359] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.431730 1178 raft_consensus.cc:385] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.431777 1178 raft_consensus.cc:740] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.431981 1178 consensus_queue.cc:260] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.432093 1946 leader_election.cc:290] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:16.433060 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:16.433233 782 raft_consensus.cc:2468] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:16.433580 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.434333 861 leader_election.cc:304] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:16.434598 1946 raft_consensus.cc:2804] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 08:52:16.434640 861 leader_election.cc:343] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.434680 1946 raft_consensus.cc:493] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:16.434741 1946 raft_consensus.cc:3060] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.437332 1946 raft_consensus.cc:515] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.437948 1946 leader_election.cc:290] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:16.440399 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:16.440749 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e1084a2b4c44c72a5904cf71b9e19af consensus state has no leader
I20251029 08:52:16.440853 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:16.441011 782 raft_consensus.cc:3060] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:16.444624 861 leader_election.cc:343] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.445389 1178 ts_tablet_manager.cc:1434] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251029 08:52:16.445719 1178 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.447317 1178 tablet_bootstrap.cc:654] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.447991 1177 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.448182 1177 ts_tablet_manager.cc:1403] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.196s user 0.023s sys 0.032s
I20251029 08:52:16.449009 1177 raft_consensus.cc:359] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.449191 1177 raft_consensus.cc:385] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.449251 1177 raft_consensus.cc:740] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.449491 1177 consensus_queue.cc:260] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.449985 1177 ts_tablet_manager.cc:1434] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:16.450279 1177 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.452625 782 raft_consensus.cc:2468] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:16.453109 861 leader_election.cc:304] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:16.453382 1946 raft_consensus.cc:2804] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:16.453505 1946 raft_consensus.cc:697] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:16.453733 1946 consensus_queue.cc:237] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.457054 1177 tablet_bootstrap.cc:654] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.461860 1178 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.462944 1178 ts_tablet_manager.cc:1403] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20251029 08:52:16.463829 1178 raft_consensus.cc:359] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.464003 1178 raft_consensus.cc:385] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.464085 1178 raft_consensus.cc:740] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.464342 1178 consensus_queue.cc:260] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.468755 1178 ts_tablet_manager.cc:1434] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 08:52:16.472303 1178 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.473418 1937 raft_consensus.cc:493] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.473551 1937 raft_consensus.cc:515] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.474148 1937 leader_election.cc:290] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.474825 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.475436 1178 tablet_bootstrap.cc:654] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.475641 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:16.476315 718 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:16.477064 720 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.477190 720 leader_election.cc:304] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.477662 1937 raft_consensus.cc:2749] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.482009 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9f4e595ea8564e9d902458aff13d5c78" } modify_external_catalogs: true
I20251029 08:52:16.482254 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9f4e595ea8564e9d902458aff13d5c78" } modify_external_catalogs: true
I20251029 08:52:16.486394 1178 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.486536 1178 ts_tablet_manager.cc:1403] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20251029 08:52:16.487264 1178 raft_consensus.cc:359] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.487442 1178 raft_consensus.cc:385] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.487494 1178 raft_consensus.cc:740] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.487677 1178 consensus_queue.cc:260] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.488160 1178 ts_tablet_manager.cc:1434] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:16.488473 1178 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.488408 1928 raft_consensus.cc:493] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.488582 1928 raft_consensus.cc:515] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.489250 1928 leader_election.cc:290] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.490242 1178 tablet_bootstrap.cc:654] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.490386 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:16.490253 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63dfcd944c1f487b8911ae383770d930" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:16.492379 998 leader_election.cc:304] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.492914 1928 raft_consensus.cc:2749] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.496570 1183 tablet_bootstrap.cc:492] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.496722 1183 ts_tablet_manager.cc:1403] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.236s user 0.036s sys 0.033s
I20251029 08:52:16.499336 1183 raft_consensus.cc:359] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.499565 1183 raft_consensus.cc:385] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.499629 1183 raft_consensus.cc:740] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.499863 1183 consensus_queue.cc:260] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.500169 1751 raft_consensus.cc:493] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.500273 1751 raft_consensus.cc:515] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.500383 1183 ts_tablet_manager.cc:1434] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:16.500628 1183 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.500844 1751 leader_election.cc:290] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.501502 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:16.502270 1183 tablet_bootstrap.cc:654] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.502672 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d11b75de7d448c68fd63ab1ae81a464" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:16.502971 782 raft_consensus.cc:2393] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:16.503562 998 leader_election.cc:304] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.504113 1751 raft_consensus.cc:3060] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.507120 1033 tablet_service.cc:1505] Processing CreateTablet for tablet a840d5bb0c374f66aac572ffc08008b3 (DEFAULT_TABLE table=default.Table_09655491a5f442aaa477ef4bbb1303c1 [id=ca14df4f74e14058939a09b591e81b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.507526 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a840d5bb0c374f66aac572ffc08008b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.507737 1751 raft_consensus.cc:2749] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.508905 762 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d97ee1ab84f9ba23946941938159c (DEFAULT_TABLE table=default.Table_4f3ecd8febb64821a555373de62d6b6e [id=201a9d9dafaf4ecd9c28ff56c44f57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.509414 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d97ee1ab84f9ba23946941938159c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.512393 756 tablet_service.cc:1505] Processing CreateTablet for tablet a840d5bb0c374f66aac572ffc08008b3 (DEFAULT_TABLE table=default.Table_09655491a5f442aaa477ef4bbb1303c1 [id=ca14df4f74e14058939a09b591e81b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.512562 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 71612f14646f421098370d81ec40adb1 (DEFAULT_TABLE table=default.Table_2f33511f2bcc46a0a692fa6f7827a66f [id=d8c2cd58577c4c35b895ab3f2e97d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.512898 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71612f14646f421098370d81ec40adb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.512899 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a840d5bb0c374f66aac572ffc08008b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:16.515275 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 51557.1us
I20251029 08:52:16.521214 1609 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 08:52:16.508905 763 tablet_service.cc:1505] Processing CreateTablet for tablet 71612f14646f421098370d81ec40adb1 (DEFAULT_TABLE table=default.Table_2f33511f2bcc46a0a692fa6f7827a66f [id=d8c2cd58577c4c35b895ab3f2e97d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.524610 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71612f14646f421098370d81ec40adb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:16.531770 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4200db822deb4ace936f561c230ac1fb consensus state has no leader
I20251029 08:52:16.532330 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5038a1ad6f534f59a938f5b34e5e27db" } modify_external_catalogs: true
I20251029 08:52:16.532553 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5038a1ad6f534f59a938f5b34e5e27db" } modify_external_catalogs: true
I20251029 08:52:16.539517 896 tablet_service.cc:1505] Processing CreateTablet for tablet a840d5bb0c374f66aac572ffc08008b3 (DEFAULT_TABLE table=default.Table_09655491a5f442aaa477ef4bbb1303c1 [id=ca14df4f74e14058939a09b591e81b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.539901 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a840d5bb0c374f66aac572ffc08008b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.540460 888 tablet_service.cc:1505] Processing CreateTablet for tablet 71612f14646f421098370d81ec40adb1 (DEFAULT_TABLE table=default.Table_2f33511f2bcc46a0a692fa6f7827a66f [id=d8c2cd58577c4c35b895ab3f2e97d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.543926 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71612f14646f421098370d81ec40adb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.553138 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_d4b6d93a144541b3b24624df6e5a0201 [id=821160def5624241afdce0f734ef2411]). Expected version 3 got 0
W20251029 08:52:16.553337 594 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58367 (request call id 104) took 928 ms (client timeout 1000 ms). Trace:
I20251029 08:52:16.554795 597 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:16.554988 597 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
W20251029 08:52:16.556740 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 consensus state has no leader
I20251029 08:52:16.557425 1178 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.557610 1178 ts_tablet_manager.cc:1403] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.036s
I20251029 08:52:16.558027 1070 raft_consensus.cc:3060] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.558609 1178 raft_consensus.cc:359] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.558802 1178 raft_consensus.cc:385] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.558847 1178 raft_consensus.cc:740] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.559051 1178 consensus_queue.cc:260] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.559595 1177 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.559712 1177 ts_tablet_manager.cc:1403] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.012s
I20251029 08:52:16.559720 1178 ts_tablet_manager.cc:1434] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.559966 1178 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.561756 1178 tablet_bootstrap.cc:654] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.564805 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.565121 899 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.565272 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.565424 894 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.565691 1177 raft_consensus.cc:359] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.565842 1177 raft_consensus.cc:385] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.565886 1177 raft_consensus.cc:740] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.566666 1177 consensus_queue.cc:260] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:16.553468 594 rpcz_store.cc:269] 1029 08:52:15.624886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:15.669964 (+ 45078us) service_pool.cc:225] Handling call
1029 08:52:16.553231 (+883267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:16.485452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:16.485590 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:16.485597 (+ 7us) write_op.cc:435] Acquired schema lock
1029 08:52:16.485601 (+ 4us) tablet.cc:660] Decoding operations
1029 08:52:16.485728 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:16.485740 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:16.485747 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:16.485860 (+ 113us) tablet.cc:699] Row locks acquired
1029 08:52:16.485864 (+ 4us) write_op.cc:260] PREPARE: finished
1029 08:52:16.485899 (+ 35us) write_op.cc:270] Start()
1029 08:52:16.485933 (+ 34us) write_op.cc:276] Timestamp: P: 1761727936485839 usec, L: 0
1029 08:52:16.485939 (+ 6us) op_driver.cc:348] REPLICATION: starting
1029 08:52:16.486114 (+ 175us) log.cc:844] Serialized 901 byte log entry
1029 08:52:16.549144 (+ 63030us) op_driver.cc:464] REPLICATION: finished
1029 08:52:16.549492 (+ 348us) write_op.cc:301] APPLY: starting
1029 08:52:16.549511 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:16.549568 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:16.549637 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:16.549727 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:16.549767 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 08:52:16.549778 (+ 11us) write_op.cc:312] APPLY: finished
1029 08:52:16.549870 (+ 92us) log.cc:844] Serialized 29 byte log entry
1029 08:52:16.549938 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:16.549960 (+ 22us) write_op.cc:454] Released schema lock
1029 08:52:16.549967 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:16.550038 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:16.550042 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":285,"apply.run_cpu_time_us":543,"apply.run_wall_time_us":543,"num_ops":2,"prepare.queue_time_us":366101,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":1207,"replication_time_us":63153}]]}
I20251029 08:52:16.567978 1177 ts_tablet_manager.cc:1434] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:16.568339 1177 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.573594 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_14f731cef6024ab79260c04bef01d3b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.574196 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14f731cef6024ab79260c04bef01d3b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.575455 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d178fa8200d645e19cfaa9177254d5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.575968 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d178fa8200d645e19cfaa9177254d5e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.578895 902 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d97ee1ab84f9ba23946941938159c (DEFAULT_TABLE table=default.Table_4f3ecd8febb64821a555373de62d6b6e [id=201a9d9dafaf4ecd9c28ff56c44f57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.579056 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0b00cd7c457d4204965de864c88be5e6" } modify_external_catalogs: true
I20251029 08:52:16.579231 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0b00cd7c457d4204965de864c88be5e6" } modify_external_catalogs: true
I20251029 08:52:16.581770 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e" } modify_external_catalogs: true
I20251029 08:52:16.581961 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc3c095bbcc142c196d51cf6e9f0fa5e" } modify_external_catalogs: true
I20251029 08:52:16.582890 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
W20251029 08:52:16.585635 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 026eef7a3cba40d0a2c090485cea55e0 consensus state has no leader
I20251029 08:52:16.589063 1183 tablet_bootstrap.cc:492] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.589236 1183 ts_tablet_manager.cc:1403] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.005s
I20251029 08:52:16.589527 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 449ffc678b874871a672c3059235de6a
I20251029 08:52:16.590044 1183 raft_consensus.cc:359] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.590211 1183 raft_consensus.cc:385] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.590261 1183 raft_consensus.cc:740] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.590448 1183 consensus_queue.cc:260] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.590569 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 308da528313044929b66040d6659f774
I20251029 08:52:16.590977 1183 ts_tablet_manager.cc:1434] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:16.591254 1183 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.592818 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d97ee1ab84f9ba23946941938159c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.594133 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_e4e823c7272f44e1af831ef60277d505 [id=93bd45834be24532b19ed170c43ac0dc]). Expected version 1 got 0
I20251029 08:52:16.594594 1177 tablet_bootstrap.cc:654] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.595685 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.596308 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.597548 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_713096b122044a3884d7d40bf3d84cb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.598003 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_713096b122044a3884d7d40bf3d84cb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.604624 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ffc678b874871a672c3059235de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.604902 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 308da528313044929b66040d6659f774 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:16.605918 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 449ffc678b874871a672c3059235de6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 449ffc678b874871a672c3059235de6a already in progress: creating tablet
I20251029 08:52:16.605986 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
W20251029 08:52:16.606263 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 308da528313044929b66040d6659f774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308da528313044929b66040d6659f774 already in progress: creating tablet
I20251029 08:52:16.606240 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
I20251029 08:52:16.607313 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 308da528313044929b66040d6659f774 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.607371 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.607769 1176 tablet_replica.cc:333] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.607965 1176 raft_consensus.cc:2243] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.608152 1176 raft_consensus.cc:2272] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.608347 762 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ffc678b874871a672c3059235de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.608564 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.608829 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.608928 761 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.609042 1176 ts_tablet_manager.cc:1916] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:16.609671 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 308da528313044929b66040d6659f774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308da528313044929b66040d6659f774 already in progress: creating tablet
I20251029 08:52:16.610698 1183 tablet_bootstrap.cc:654] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.610813 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 308da528313044929b66040d6659f774 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.611022 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ffc678b874871a672c3059235de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.609189 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.609361 747 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.616134 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.617089 1176 ts_tablet_manager.cc:1929] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.617249 1176 log.cc:1199] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:16.617117 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.617684 1176 ts_tablet_manager.cc:1950] T 787d6d7f1870450fa21cb9cba34e72fe P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:16.618172 595 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 19c515944ecf4228a7cca63cfd3565d8 consensus state has no leader
I20251029 08:52:16.618956 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 787d6d7f1870450fa21cb9cba34e72fe (table default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]) successfully deleted
I20251029 08:52:16.619320 1176 tablet_replica.cc:333] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.619552 1176 raft_consensus.cc:2243] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.619729 1176 raft_consensus.cc:2272] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.628314 588 catalog_manager.cc:5649] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:16.629132 588 catalog_manager.cc:5649] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:16.629583 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:16.630046 1183 tablet_bootstrap.cc:492] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.630193 1183 ts_tablet_manager.cc:1403] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.003s
I20251029 08:52:16.630059 588 catalog_manager.cc:5649] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:16.630653 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:16.630899 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
I20251029 08:52:16.630951 1183 raft_consensus.cc:359] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.631069 1176 ts_tablet_manager.cc:1916] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.631114 1183 raft_consensus.cc:385] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.631160 1183 raft_consensus.cc:740] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.632493 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet dee931aa035748d08b1260e8ab3fca5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.632736 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 8daa66608cf94913afb94ef661b3c898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.632907 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet fa8884bb8e64474d936c8825d1ef3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.633104 1179 tablet_replica.cc:333] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.633098 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:16.633342 1179 raft_consensus.cc:2243] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.633507 1179 raft_consensus.cc:2272] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.634353 1179 ts_tablet_manager.cc:1916] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.634645 1183 consensus_queue.cc:260] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.635139 1183 ts_tablet_manager.cc:1434] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 08:52:16.635387 1183 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.637037 1183 tablet_bootstrap.cc:654] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.637588 1974 tablet_replica.cc:333] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.637781 1974 raft_consensus.cc:2243] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.638857 1974 raft_consensus.cc:2272] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.639387 1974 ts_tablet_manager.cc:1916] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.644145 1937 raft_consensus.cc:493] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.644282 1937 raft_consensus.cc:515] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.644909 1937 leader_election.cc:290] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.645694 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.646255 720 leader_election.cc:343] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.646768 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:16.647368 718 leader_election.cc:343] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.647485 718 leader_election.cc:304] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.647886 1937 raft_consensus.cc:2749] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.652349 1178 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.652495 1178 ts_tablet_manager.cc:1403] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.012s
I20251029 08:52:16.653344 1178 raft_consensus.cc:359] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.653513 1178 raft_consensus.cc:385] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.653568 1178 raft_consensus.cc:740] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.655434 1176 ts_tablet_manager.cc:1929] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.655546 1176 log.cc:1199] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/072cf27b0743488c8166e03b5e61c545
I20251029 08:52:16.655864 1176 ts_tablet_manager.cc:1950] T 072cf27b0743488c8166e03b5e61c545 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:16.656837 1176 tablet_replica.cc:333] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.657004 1176 raft_consensus.cc:2243] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.657033 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 072cf27b0743488c8166e03b5e61c545 (table default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]) successfully deleted
I20251029 08:52:16.657152 1176 raft_consensus.cc:2272] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.658066 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fc78a202be244c3e9ad354bead9bfe52
I20251029 08:52:16.658807 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet fc78a202be244c3e9ad354bead9bfe52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.658850 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e2f8b7294ccd4916b90086595173eb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.659404 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2f8b7294ccd4916b90086595173eb9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.659426 756 tablet_service.cc:1552] Processing DeleteTablet for tablet fc78a202be244c3e9ad354bead9bfe52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.659868 895 tablet_service.cc:1552] Processing DeleteTablet for tablet fc78a202be244c3e9ad354bead9bfe52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.659982 1178 consensus_queue.cc:260] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.661469 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6773a40abf1c451bbba19564445118ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.662000 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6773a40abf1c451bbba19564445118ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.662880 1974 ts_tablet_manager.cc:1929] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.662986 1974 log.cc:1199] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/74fe974a368a4afe8a5da6dfd1036400
I20251029 08:52:16.663144 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3eda8a3eed924142880c9b465eb759b9" } new_table_name: "default.Table_5018010ef9af4f16b6388715c7f9fd03" modify_external_catalogs: true
I20251029 08:52:16.663448 1974 ts_tablet_manager.cc:1950] T 74fe974a368a4afe8a5da6dfd1036400 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.664036 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 74fe974a368a4afe8a5da6dfd1036400 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432]) successfully deleted
I20251029 08:52:16.664417 1974 tablet_replica.cc:333] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.664660 1974 raft_consensus.cc:2243] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.664867 1974 raft_consensus.cc:2272] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.665428 1974 ts_tablet_manager.cc:1916] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.666920 1179 ts_tablet_manager.cc:1929] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.667080 1179 log.cc:1199] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/dee931aa035748d08b1260e8ab3fca5c
I20251029 08:52:16.667436 1179 ts_tablet_manager.cc:1950] T dee931aa035748d08b1260e8ab3fca5c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.668742 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet dee931aa035748d08b1260e8ab3fca5c (table default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]) successfully deleted
I20251029 08:52:16.671535 1177 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.671685 1177 ts_tablet_manager.cc:1403] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.012s
I20251029 08:52:16.676867 1177 raft_consensus.cc:359] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.677047 1177 raft_consensus.cc:385] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.677086 1177 raft_consensus.cc:740] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.677253 1177 consensus_queue.cc:260] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.680224 1179 tablet_replica.cc:333] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.680131 1177 ts_tablet_manager.cc:1434] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:16.680454 1179 raft_consensus.cc:2243] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.680680 1179 raft_consensus.cc:2272] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:16.680879 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu pid 1262: Timed out: CheckLeaks RPC to 127.0.104.62:40435 timed out after 1.000s (SENT)
I20251029 08:52:16.681071 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu with pid 1262
I20251029 08:52:16.681310 1176 ts_tablet_manager.cc:1916] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.684612 1179 ts_tablet_manager.cc:1916] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.688282 591 master_service.cc:954] ReplaceTablet: received request to replace tablet c1f6ca7d4e534a46a1fba75acd7df39e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.690444 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.692081 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 64058f9cb3ad4844ac87bc0cc27774b9
I20251029 08:52:16.692466 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 24834985a71349b9865840e9b5537d72
I20251029 08:52:16.694865 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.695111 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.695339 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.695562 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.696386 1177 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.696482 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 2723cdd513ee4e8ca6773b38f785b09e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.698302 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 64058f9cb3ad4844ac87bc0cc27774b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.698444 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.699000 1183 tablet_bootstrap.cc:492] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.699126 1183 ts_tablet_manager.cc:1403] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.004s
I20251029 08:52:16.699867 1183 raft_consensus.cc:359] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.700050 1183 raft_consensus.cc:385] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.700099 1183 raft_consensus.cc:740] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.700287 1183 consensus_queue.cc:260] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.702701 1183 ts_tablet_manager.cc:1434] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:16.702991 1183 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.704999 1178 ts_tablet_manager.cc:1434] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20251029 08:52:16.706992 1178 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.708084 1183 tablet_bootstrap.cc:654] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.708963 1178 tablet_bootstrap.cc:654] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.711889 1979 raft_consensus.cc:493] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.712127 1177 tablet_bootstrap.cc:654] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.712067 1979 raft_consensus.cc:515] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.713111 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07906300f1d54d4e8fde5165ee14c44f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.713682 720 leader_election.cc:343] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.714470 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:16.714495 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07906300f1d54d4e8fde5165ee14c44f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:16.715029 718 leader_election.cc:343] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.715137 1974 ts_tablet_manager.cc:1929] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.715142 718 leader_election.cc:304] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.715600 1937 raft_consensus.cc:2749] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.715667 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 79a3ef8389944c32bba7b3e598824118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.715811 1979 leader_election.cc:290] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.717253 1179 ts_tablet_manager.cc:1929] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.717382 1179 log.cc:1199] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/8daa66608cf94913afb94ef661b3c898
I20251029 08:52:16.717681 1179 ts_tablet_manager.cc:1950] T 8daa66608cf94913afb94ef661b3c898 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.715230 1974 log.cc:1199] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/dee931aa035748d08b1260e8ab3fca5c
I20251029 08:52:16.718741 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 8daa66608cf94913afb94ef661b3c898 (table default.Table_b3fef48b968b4888a66108a7e79d0ef7 [id=f0bec519be3441c9bc5bd5b3188bc373]) successfully deleted
I20251029 08:52:16.719064 1179 tablet_replica.cc:333] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.719260 1179 raft_consensus.cc:2243] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.719377 895 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.719413 1179 raft_consensus.cc:2272] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.719713 896 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.719872 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.719923 1179 ts_tablet_manager.cc:1916] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.720772 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.720966 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.721213 902 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.721810 1974 ts_tablet_manager.cc:1950] T dee931aa035748d08b1260e8ab3fca5c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.723733 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet dee931aa035748d08b1260e8ab3fca5c (table default.Table_9ccaa6622c504f60b75e84efc44f6890 [id=457c22113c9f4f949269579ee9c31a70]) successfully deleted
I20251029 08:52:16.725965 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.726303 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.728525 1178 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.728760 1178 ts_tablet_manager.cc:1403] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20251029 08:52:16.729709 1178 raft_consensus.cc:359] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.729990 1178 raft_consensus.cc:385] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.730077 1178 raft_consensus.cc:740] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.730352 1178 consensus_queue.cc:260] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.730975 1178 ts_tablet_manager.cc:1434] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:16.731160 1974 tablet_replica.cc:333] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.731384 1974 raft_consensus.cc:2243] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.731727 1178 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.732932 1974 raft_consensus.cc:2272] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.733587 1974 ts_tablet_manager.cc:1916] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.734095 1183 tablet_bootstrap.cc:492] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.734233 1183 ts_tablet_manager.cc:1403] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20251029 08:52:16.734817 1979 raft_consensus.cc:493] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.734886 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b7ef7d2c0a184dd78ade540aa0d5cd81" } new_table_name: "default.Table_be549a784efc4332bf662c497d35ce49" modify_external_catalogs: true
I20251029 08:52:16.734946 1979 raft_consensus.cc:515] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:16.735013 1183 raft_consensus.cc:359] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.735159 1183 raft_consensus.cc:385] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.735216 1183 raft_consensus.cc:740] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.735462 1178 tablet_bootstrap.cc:654] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.735414 1183 consensus_queue.cc:260] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.735905 1183 ts_tablet_manager.cc:1434] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:16.736088 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.736610 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 116cc0d155e24653937953eea4f39ee4 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.736908 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:16.737588 720 leader_election.cc:343] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:16.737859 718 leader_election.cc:343] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.737954 718 leader_election.cc:304] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.738019 1183 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.738175 1937 raft_consensus.cc:2749] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.735541 1979 leader_election.cc:290] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:16.739511 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c2e24ece1404009b4b7765c555b549f" } new_table_name: "default.Table_31cece98e77343e28a420add5c5d9c0e" modify_external_catalogs: true
I20251029 08:52:16.741315 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fadb0daa1e404b088a08bc134f4bf4a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.741642 1177 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.741761 1177 ts_tablet_manager.cc:1403] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.002s
W20251029 08:52:16.741885 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fadb0daa1e404b088a08bc134f4bf4a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.742580 1177 raft_consensus.cc:359] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.742753 1177 raft_consensus.cc:385] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.742803 1177 raft_consensus.cc:740] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.743052 1177 consensus_queue.cc:260] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.743656 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f40b8aaae0b34603be167b08d72d08d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.744475 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40b8aaae0b34603be167b08d72d08d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.744709 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_04ec7a02b4e44860b339d5e95468a084 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 1 got 0
I20251029 08:52:16.744911 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:16.745678 1177 ts_tablet_manager.cc:1434] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:16.745968 1177 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.746181 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 8626877e3cfc4cb48f6caa92638b4917 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.747519 1177 tablet_bootstrap.cc:654] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.750180 1974 ts_tablet_manager.cc:1929] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.752383 1974 log.cc:1199] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/3295bb0b47714080a5c3f34049acd487
I20251029 08:52:16.752933 1974 ts_tablet_manager.cc:1950] T 3295bb0b47714080a5c3f34049acd487 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.753674 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.755043 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 3295bb0b47714080a5c3f34049acd487 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
I20251029 08:52:16.755996 1183 tablet_bootstrap.cc:654] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.756875 1178 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.757074 1178 ts_tablet_manager.cc:1403] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20251029 08:52:16.757647 1179 ts_tablet_manager.cc:1929] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.757761 1179 log.cc:1199] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/fa8884bb8e64474d936c8825d1ef3545
I20251029 08:52:16.758051 1179 ts_tablet_manager.cc:1950] T fa8884bb8e64474d936c8825d1ef3545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.757999 1178 raft_consensus.cc:359] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.758284 1178 raft_consensus.cc:385] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.758404 1178 raft_consensus.cc:740] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.758704 1178 consensus_queue.cc:260] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.759003 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet fa8884bb8e64474d936c8825d1ef3545 (table default.Table_d3b4e5bde9434116b192d820a909c7b1 [id=064ab59757e94a9089fed2c7b3425476]) successfully deleted
I20251029 08:52:16.759284 1178 ts_tablet_manager.cc:1434] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:16.759646 1178 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.760052 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4f68e8291cd64391bdb236f948dd338b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.760684 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f68e8291cd64391bdb236f948dd338b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.762140 1178 tablet_bootstrap.cc:654] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.762391 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 74fe974a368a4afe8a5da6dfd1036400 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.763123 1974 tablet_replica.cc:333] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.763379 1974 raft_consensus.cc:2243] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.763612 1974 raft_consensus.cc:2272] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.764192 1974 ts_tablet_manager.cc:1916] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.764763 1177 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.764884 1177 ts_tablet_manager.cc:1403] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251029 08:52:16.765509 1177 raft_consensus.cc:359] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.765676 1177 raft_consensus.cc:385] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.765733 1177 raft_consensus.cc:740] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.765966 1177 consensus_queue.cc:260] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.769016 1177 ts_tablet_manager.cc:1434] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:16.769304 1177 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.770254 1979 raft_consensus.cc:493] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.770401 1979 raft_consensus.cc:515] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.771019 1979 leader_election.cc:290] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.771353 1178 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.771530 1178 ts_tablet_manager.cc:1403] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20251029 08:52:16.771660 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "840ff254244f43a18679232279e02d1f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.772405 1178 raft_consensus.cc:359] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.772529 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "840ff254244f43a18679232279e02d1f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:16.772626 1178 raft_consensus.cc:385] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.772691 1178 raft_consensus.cc:740] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.772903 1178 consensus_queue.cc:260] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:16.773155 720 leader_election.cc:343] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.773505 1178 ts_tablet_manager.cc:1434] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251029 08:52:16.773658 718 leader_election.cc:343] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.773765 718 leader_election.cc:304] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.774140 1979 raft_consensus.cc:2749] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.773758 1178 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.776533 1178 tablet_bootstrap.cc:654] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:16.778968 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fc78a202be244c3e9ad354bead9bfe52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc78a202be244c3e9ad354bead9bfe52 already in progress: creating tablet
W20251029 08:52:16.779660 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 64058f9cb3ad4844ac87bc0cc27774b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64058f9cb3ad4844ac87bc0cc27774b9 already in progress: creating tablet
W20251029 08:52:16.780884 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 24834985a71349b9865840e9b5537d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24834985a71349b9865840e9b5537d72 already in progress: creating tablet
I20251029 08:52:16.780951 1177 tablet_bootstrap.cc:654] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.781235 1179 tablet_replica.cc:333] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.781504 1179 raft_consensus.cc:2243] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:16.783408 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d26ff78aa5f94ae49a197daf2be00aa5 consensus state has no leader
I20251029 08:52:16.783804 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_159d7048427049ffb8b14f34b6bb78fa" } new_table_name: "default.Table_3f1b48395900452b8b149c6aef010592" modify_external_catalogs: true
I20251029 08:52:16.784490 1177 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.784613 1177 ts_tablet_manager.cc:1403] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20251029 08:52:16.785140 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c1f6ca7d4e534a46a1fba75acd7df39e
I20251029 08:52:16.785223 1177 raft_consensus.cc:359] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.785364 1177 raft_consensus.cc:385] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.785405 1177 raft_consensus.cc:740] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.785567 591 catalog_manager.cc:6719] ReplaceTablet: tablet c1f6ca7d4e534a46a1fba75acd7df39e deleted and replaced by tablet 62b7fe26c7ad48cdb935e8dc6b1962a7
I20251029 08:52:16.785825 591 master_service.cc:954] ReplaceTablet: received request to replace tablet eeabf8f23751447aa9af02363c40361d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.786895 902 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e with delete_type TABLET_DATA_DELETED (replaced by tablet 62b7fe26c7ad48cdb935e8dc6b1962a7) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:16.787120 1177 consensus_queue.cc:260] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.787240 747 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e with delete_type TABLET_DATA_DELETED (replaced by tablet 62b7fe26c7ad48cdb935e8dc6b1962a7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.787359 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f6ca7d4e534a46a1fba75acd7df39e with delete_type TABLET_DATA_DELETED (replaced by tablet 62b7fe26c7ad48cdb935e8dc6b1962a7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.787778 1179 raft_consensus.cc:2272] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.788714 1179 ts_tablet_manager.cc:1916] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.789412 1177 ts_tablet_manager.cc:1434] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:16.789692 1177 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.789786 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a1a3d90a54344467a739bc9d4d5a1b91" } new_table_name: "default.Table_78e22440874d4598ac910219a652f3ed" modify_external_catalogs: true
I20251029 08:52:16.791091 1979 raft_consensus.cc:493] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.791316 1177 tablet_bootstrap.cc:654] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.791232 1979 raft_consensus.cc:515] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.791878 1979 leader_election.cc:290] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.792512 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fcf8a3e0f4770a9743fddffa6495f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.793154 720 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.793700 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fcf8a3e0f4770a9743fddffa6495f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:16.794258 718 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.794389 718 leader_election.cc:304] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.794873 1979 raft_consensus.cc:2749] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.797693 747 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.799643 1974 ts_tablet_manager.cc:1929] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.799887 1974 log.cc:1199] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c9a1d5e1f68e4f0e8b78641bbdc98123
I20251029 08:52:16.800282 1974 ts_tablet_manager.cc:1950] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.801326 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:16.806545 1946 raft_consensus.cc:493] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.806690 1946 raft_consensus.cc:515] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.807288 1946 leader_election.cc:290] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.808110 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c5008d9df9419bb4b6d678c6dc27a6" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.808593 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 308da528313044929b66040d6659f774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308da528313044929b66040d6659f774 already in progress: creating tablet
W20251029 08:52:16.808640 861 leader_election.cc:343] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c5008d9df9419bb4b6d678c6dc27a6
W20251029 08:52:16.808833 861 leader_election.cc:343] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.808914 861 leader_election.cc:304] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:16.809190 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 449ffc678b874871a672c3059235de6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 449ffc678b874871a672c3059235de6a already in progress: creating tablet
I20251029 08:52:16.809295 1946 raft_consensus.cc:2749] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.810710 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c8619310a8224be58710ffee5aa94c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.810842 1183 tablet_bootstrap.cc:492] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.810967 1183 ts_tablet_manager.cc:1403] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.005s
I20251029 08:52:16.810984 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79ee6c7183a04b5c93eb9a318008b975" } modify_external_catalogs: true
I20251029 08:52:16.811143 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79ee6c7183a04b5c93eb9a318008b975" } modify_external_catalogs: true
W20251029 08:52:16.811282 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8619310a8224be58710ffee5aa94c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.811693 1183 raft_consensus.cc:359] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.811846 1183 raft_consensus.cc:385] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.811905 1183 raft_consensus.cc:740] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.812148 1183 consensus_queue.cc:260] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:16.812278 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87c98206d7fa409b8017abc72688db93 consensus state has no leader
I20251029 08:52:16.812698 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13df179414624614b1663578c3d515be" } modify_external_catalogs: true
I20251029 08:52:16.812875 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13df179414624614b1663578c3d515be" } modify_external_catalogs: true
I20251029 08:52:16.813376 1183 ts_tablet_manager.cc:1434] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:16.813426 1974 tablet_replica.cc:333] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.813594 1974 raft_consensus.cc:2243] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.813688 1183 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.813958 1974 raft_consensus.cc:2272] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.814698 1974 ts_tablet_manager.cc:1916] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.815256 1178 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.815400 1183 tablet_bootstrap.cc:654] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.815413 1178 ts_tablet_manager.cc:1403] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20251029 08:52:16.816205 1178 raft_consensus.cc:359] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.816408 1178 raft_consensus.cc:385] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.816501 1178 raft_consensus.cc:740] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.816740 1178 consensus_queue.cc:260] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.817489 1178 ts_tablet_manager.cc:1434] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:16.817824 1178 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.820111 1178 tablet_bootstrap.cc:654] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:16.821795 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 consensus state has no leader
I20251029 08:52:16.824796 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 3295bb0b47714080a5c3f34049acd487 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:16.826148 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.826337 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.827479 1974 ts_tablet_manager.cc:1929] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.827672 1974 log.cc:1199] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/79a3ef8389944c32bba7b3e598824118
I20251029 08:52:16.827764 1176 ts_tablet_manager.cc:1929] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.827863 1176 log.cc:1199] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:16.828282 1974 ts_tablet_manager.cc:1950] T 79a3ef8389944c32bba7b3e598824118 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.828337 1179 ts_tablet_manager.cc:1929] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.828500 1179 log.cc:1199] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/79a3ef8389944c32bba7b3e598824118
I20251029 08:52:16.828878 1179 ts_tablet_manager.cc:1950] T 79a3ef8389944c32bba7b3e598824118 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.829639 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 79a3ef8389944c32bba7b3e598824118 (table default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]) successfully deleted
I20251029 08:52:16.830646 1177 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.830794 1177 ts_tablet_manager.cc:1403] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20251029 08:52:16.830979 1974 tablet_replica.cc:333] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.831233 1974 raft_consensus.cc:2243] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.831436 1974 raft_consensus.cc:2272] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.831399 1179 tablet_replica.cc:333] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.831825 1179 raft_consensus.cc:2243] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.832146 1179 raft_consensus.cc:2272] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.832582 1974 ts_tablet_manager.cc:1916] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.830011 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 79a3ef8389944c32bba7b3e598824118 (table default.Table_1087b19fb3a74025b8269c7eb247bc14 [id=f022d5c1585c47289b0157c47a8da026]) successfully deleted
I20251029 08:52:16.833422 1179 ts_tablet_manager.cc:1916] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.834098 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet 757e34137e884d5ca5cd88fdd03244d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:16.831547 1177 raft_consensus.cc:359] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.836088 1177 raft_consensus.cc:385] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.836163 1177 raft_consensus.cc:740] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.836424 1177 consensus_queue.cc:260] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.838443 1176 ts_tablet_manager.cc:1950] T 608b7095d42941ccb0c4eb57f6754e74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:16.839370 1177 ts_tablet_manager.cc:1434] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 08:52:16.839736 1177 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.839934 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 608b7095d42941ccb0c4eb57f6754e74 (table default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]) successfully deleted
I20251029 08:52:16.840788 1183 tablet_bootstrap.cc:492] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.840907 1183 ts_tablet_manager.cc:1403] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20251029 08:52:16.843144 1183 raft_consensus.cc:359] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.843385 1183 raft_consensus.cc:385] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.843461 1183 raft_consensus.cc:740] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.843750 1183 consensus_queue.cc:260] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.844338 1183 ts_tablet_manager.cc:1434] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:16.847142 1183 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.850241 1183 tablet_bootstrap.cc:654] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.851300 1177 tablet_bootstrap.cc:654] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.846904 1176 tablet_replica.cc:333] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:16.850171 1178 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.854507 1178 ts_tablet_manager.cc:1403] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.005s
I20251029 08:52:16.855408 1178 raft_consensus.cc:359] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.855597 1178 raft_consensus.cc:385] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.855665 1178 raft_consensus.cc:740] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.855885 1178 consensus_queue.cc:260] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.856478 1178 ts_tablet_manager.cc:1434] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:16.856853 1178 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.857069 1048 tablet_service.cc:1552] Processing DeleteTablet for tablet c9a1d5e1f68e4f0e8b78641bbdc98123 with delete_type TABLET_DATA_DELETED (replaced by tablet a3ba512bd4f04bd9aceb4cb210fe9689) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:16.858809 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5e1e3ce83b14817b2440370cb2fcabd consensus state has no leader
W20251029 08:52:16.859237 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cea6d85081f246788ba29bb4a9c9a4e8 has no consensus state
I20251029 08:52:16.859282 1751 raft_consensus.cc:493] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.859422 1751 raft_consensus.cc:515] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.860949 1178 tablet_bootstrap.cc:654] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.862062 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_703d8f64e5cd49e58f0354ed41369097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.862598 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_703d8f64e5cd49e58f0354ed41369097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.869518 1179 ts_tablet_manager.cc:1929] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.869657 1179 log.cc:1199] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/8626877e3cfc4cb48f6caa92638b4917
I20251029 08:52:16.869975 1179 ts_tablet_manager.cc:1950] T 8626877e3cfc4cb48f6caa92638b4917 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:16.871796 719 connection.cc:537] client connection to 127.0.104.62:40435 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 08:52:16.872633 1183 tablet_bootstrap.cc:492] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.872783 1183 ts_tablet_manager.cc:1403] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.006s
I20251029 08:52:16.873844 1183 raft_consensus.cc:359] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.874058 1183 raft_consensus.cc:385] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.874119 1183 raft_consensus.cc:740] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.874366 1183 consensus_queue.cc:260] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.874964 1183 ts_tablet_manager.cc:1434] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251029 08:52:16.874966 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:40435 (error 108) [suppressed 2 similar messages]
I20251029 08:52:16.875938 1751 leader_election.cc:290] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:16.879904 998 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:16.881103 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757e34137e884d5ca5cd88fdd03244d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:16.881331 922 raft_consensus.cc:2468] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:16.881453 1183 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.882220 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.882812 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:16.883423 860 connection.cc:537] client connection to 127.0.104.62:40435 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 08:52:16.883579 1183 tablet_bootstrap.cc:654] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.884193 996 leader_election.cc:304] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.884676 1751 raft_consensus.cc:2804] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:16.884784 1751 raft_consensus.cc:493] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:16.884856 1751 raft_consensus.cc:3060] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.887688 1751 raft_consensus.cc:515] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.890859 1183 tablet_bootstrap.cc:492] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.891045 1183 ts_tablet_manager.cc:1403] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251029 08:52:16.891893 1183 raft_consensus.cc:359] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.892146 1183 raft_consensus.cc:385] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.892230 1183 raft_consensus.cc:740] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.893945 1179 tablet_replica.cc:333] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:16.894167 1179 raft_consensus.cc:2243] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.894405 1179 raft_consensus.cc:2272] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.894838 1974 ts_tablet_manager.cc:1929] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.894963 1974 log.cc:1199] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/8626877e3cfc4cb48f6caa92638b4917
I20251029 08:52:16.895371 1974 ts_tablet_manager.cc:1950] T 8626877e3cfc4cb48f6caa92638b4917 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.895534 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e66832af5e114b158d59c2326322b87c" } modify_external_catalogs: true
I20251029 08:52:16.895754 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e66832af5e114b158d59c2326322b87c" } modify_external_catalogs: true
I20251029 08:52:16.896342 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 8626877e3cfc4cb48f6caa92638b4917 (table default.Table_c43c6468d7c447b28f338ce82a1df720 [id=38e7e568f136480caaa0ee51ea6927f3]) successfully deleted
W20251029 08:52:16.896868 998 leader_election.cc:343] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:16.898381 1974 tablet_replica.cc:333] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.898571 1974 raft_consensus.cc:2243] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.898722 1974 raft_consensus.cc:2272] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.899438 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 8626877e3cfc4cb48f6caa92638b4917 (table default.Table_c43c6468d7c447b28f338ce82a1df720 [id=38e7e568f136480caaa0ee51ea6927f3]) successfully deleted
I20251029 08:52:16.899489 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_aae83ce4ba7f4a1288306bf92745e84e" } modify_external_catalogs: true
I20251029 08:52:16.899648 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_aae83ce4ba7f4a1288306bf92745e84e" } modify_external_catalogs: true
W20251029 08:52:16.899715 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fc78a202be244c3e9ad354bead9bfe52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc78a202be244c3e9ad354bead9bfe52 already in progress: creating tablet
W20251029 08:52:16.899922 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 64058f9cb3ad4844ac87bc0cc27774b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64058f9cb3ad4844ac87bc0cc27774b9 already in progress: creating tablet
W20251029 08:52:16.901000 1996 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 08:52:16.872886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40435 (local address 127.0.104.1:54421)
1029 08:52:16.900689 (+ 27803us) negotiation.cc:107] Waiting for socket to connect
1029 08:52:16.900795 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27655,"threads_started":1}
I20251029 08:52:16.901541 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a565b344ef8e4b60b07cb259f877df2a" } modify_external_catalogs: true
I20251029 08:52:16.901685 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a565b344ef8e4b60b07cb259f877df2a" } modify_external_catalogs: true
I20251029 08:52:16.901880 1178 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.902002 1178 ts_tablet_manager.cc:1403] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.004s
I20251029 08:52:16.902909 1178 raft_consensus.cc:359] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.903085 1178 raft_consensus.cc:385] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.903158 1178 raft_consensus.cc:740] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.903383 1178 consensus_queue.cc:260] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.904311 1177 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.904440 1177 ts_tablet_manager.cc:1403] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.008s
I20251029 08:52:16.905069 1177 raft_consensus.cc:359] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.905211 1177 raft_consensus.cc:385] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.905262 1177 raft_consensus.cc:740] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.905473 1177 consensus_queue.cc:260] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.907392 1177 ts_tablet_manager.cc:1434] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:16.907648 1177 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.909245 1177 tablet_bootstrap.cc:654] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.909264 1178 ts_tablet_manager.cc:1434] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20251029 08:52:16.910050 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 24834985a71349b9865840e9b5537d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24834985a71349b9865840e9b5537d72 already in progress: creating tablet
I20251029 08:52:16.911250 1183 consensus_queue.cc:260] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:16.911535 1179 ts_tablet_manager.cc:1916] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.912842 1946 raft_consensus.cc:493] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.912902 1974 ts_tablet_manager.cc:1916] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.912961 1946 raft_consensus.cc:515] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.913528 1946 leader_election.cc:290] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:16.914423 861 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:16.914800 1178 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.914775 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a31847928db44bd4b1ebd50c5130065f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.915266 861 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.915367 861 leader_election.cc:304] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.915694 1946 raft_consensus.cc:2749] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.916249 1751 leader_election.cc:290] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.916458 1178 tablet_bootstrap.cc:654] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.916638 1183 ts_tablet_manager.cc:1434] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251029 08:52:16.916965 1183 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.917414 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757e34137e884d5ca5cd88fdd03244d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:16.917562 922 raft_consensus.cc:3060] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:16.919100 1183 tablet_bootstrap.cc:654] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.919317 1178 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.919457 1178 ts_tablet_manager.cc:1403] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251029 08:52:16.920360 922 raft_consensus.cc:2468] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:16.920490 1178 raft_consensus.cc:359] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.920662 1178 raft_consensus.cc:385] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.920720 1178 raft_consensus.cc:740] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.920928 1178 consensus_queue.cc:260] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.922362 1179 ts_tablet_manager.cc:1929] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.922498 1179 log.cc:1199] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/74fe974a368a4afe8a5da6dfd1036400
I20251029 08:52:16.922549 996 leader_election.cc:304] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.922858 1179 ts_tablet_manager.cc:1950] T 74fe974a368a4afe8a5da6dfd1036400 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:16.923231 1751 raft_consensus.cc:2804] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:16.923818 1178 ts_tablet_manager.cc:1434] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:16.924256 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 74fe974a368a4afe8a5da6dfd1036400 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432]) successfully deleted
I20251029 08:52:16.924723 1751 raft_consensus.cc:697] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:16.925010 1178 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.925140 1751 consensus_queue.cc:237] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.926306 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0956bbcc718e4744ab316e8efd03b1fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.926910 1178 tablet_bootstrap.cc:654] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:16.926954 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0956bbcc718e4744ab316e8efd03b1fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.928041 1979 raft_consensus.cc:493] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.928193 1979 raft_consensus.cc:515] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.928870 1979 leader_election.cc:290] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:16.929419 1946 raft_consensus.cc:493] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.929479 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd25eeb89e574297a92d8e978a79d768" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:16.929536 1946 raft_consensus.cc:515] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.929613 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd25eeb89e574297a92d8e978a79d768" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.929934 718 leader_election.cc:343] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.930048 1946 leader_election.cc:290] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:16.930130 720 leader_election.cc:343] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.930226 720 leader_election.cc:304] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:16.930594 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215cd666c71a4179a19e298868a88e96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:16.931015 861 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.931088 1979 raft_consensus.cc:2749] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:16.931190 861 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 215cd666c71a4179a19e298868a88e96
I20251029 08:52:16.931293 861 leader_election.cc:304] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.931663 1179 tablet_replica.cc:333] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:16.931704 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c1f6ca7d4e534a46a1fba75acd7df39e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f6ca7d4e534a46a1fba75acd7df39e already in progress: creating tablet
I20251029 08:52:16.932144 1177 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.932281 1177 ts_tablet_manager.cc:1403] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20251029 08:52:16.932422 1946 raft_consensus.cc:2749] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.933173 1177 raft_consensus.cc:359] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.933375 1177 raft_consensus.cc:385] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.933435 1177 raft_consensus.cc:740] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.933650 1177 consensus_queue.cc:260] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.934193 1177 ts_tablet_manager.cc:1434] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:16.934733 1177 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.937664 1177 tablet_bootstrap.cc:654] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.939383 1178 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.939509 1178 ts_tablet_manager.cc:1403] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251029 08:52:16.940191 1178 raft_consensus.cc:359] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.940419 1178 raft_consensus.cc:385] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.940470 1178 raft_consensus.cc:740] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.940647 1178 consensus_queue.cc:260] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.943344 1178 ts_tablet_manager.cc:1434] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:16.944437 1178 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.946056 1178 tablet_bootstrap.cc:654] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.947827 1177 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.947962 1177 ts_tablet_manager.cc:1403] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20251029 08:52:16.948709 1177 raft_consensus.cc:359] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.948868 1177 raft_consensus.cc:385] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.948990 1177 raft_consensus.cc:740] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.949208 1177 consensus_queue.cc:260] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.854516 1176 raft_consensus.cc:2243] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.950587 1176 raft_consensus.cc:2272] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.951095 1177 ts_tablet_manager.cc:1434] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251029 08:52:16.951371 1177 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.953838 1176 ts_tablet_manager.cc:1916] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.955029 1946 raft_consensus.cc:493] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:16.955150 1946 raft_consensus.cc:515] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.955758 1946 leader_election.cc:290] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:16.956404 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5fa181849474a559701fdb8c18d398b" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:16.956888 1183 tablet_bootstrap.cc:492] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.957021 1183 ts_tablet_manager.cc:1403] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
W20251029 08:52:16.957443 861 leader_election.cc:343] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:16.957715 1183 raft_consensus.cc:359] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.957865 1183 raft_consensus.cc:385] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.957937 1183 raft_consensus.cc:740] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.957846 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d55bedf72b97413a8b66d1ab9a7ee591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:16.958128 1183 consensus_queue.cc:260] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:16.958473 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d55bedf72b97413a8b66d1ab9a7ee591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.958586 1183 ts_tablet_manager.cc:1434] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:16.958854 1183 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:16.958947 861 leader_election.cc:343] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e5fa181849474a559701fdb8c18d398b
I20251029 08:52:16.959039 861 leader_election.cc:304] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:16.959491 1946 raft_consensus.cc:2749] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:16.960412 1183 tablet_bootstrap.cc:654] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.964457 1177 tablet_bootstrap.cc:654] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.967093 1178 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:16.967218 1178 ts_tablet_manager.cc:1403] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20251029 08:52:16.967941 1178 raft_consensus.cc:359] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.968111 1178 raft_consensus.cc:385] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.968154 1178 raft_consensus.cc:740] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.968317 1178 consensus_queue.cc:260] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.968777 1178 ts_tablet_manager.cc:1434] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 08:52:16.969002 1178 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:16.969945 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:16.970520 1178 tablet_bootstrap.cc:654] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.970710 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2c055a320a224f7fa1d6544e49d9989f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.971184 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c055a320a224f7fa1d6544e49d9989f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.972313 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a7e2b6a295bc41f5885bd96998bd2723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:16.972826 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7e2b6a295bc41f5885bd96998bd2723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:16.973817 1974 ts_tablet_manager.cc:1929] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:16.973925 1974 log.cc:1199] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:16.974263 1974 ts_tablet_manager.cc:1950] T ba1bcb3f6c134de2b322696501927a16 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:16.974694 1974 tablet_replica.cc:333] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:16.974850 1974 raft_consensus.cc:2243] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:16.974864 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 (DEFAULT_TABLE table=default.Table_079d27f45f6a49e093d99a073678a387 [id=09890bc173754a139eec6c4eecbefddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.974989 1974 raft_consensus.cc:2272] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.975260 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8970ca1f227849a1a99ac5d33e450e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.975481 1974 ts_tablet_manager.cc:1916] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.980372 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 39bd0c8760fb4c3093c67cf01af79cc7 (DEFAULT_TABLE table=default.Table_e5472287b7984f8f8a0fbd29cee00c13 [id=87fbc692c3274f5ba176618de97f15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.980717 1177 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.980723 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd0c8760fb4c3093c67cf01af79cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.980826 1177 ts_tablet_manager.cc:1403] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251029 08:52:16.981529 1177 raft_consensus.cc:359] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.981715 1177 raft_consensus.cc:385] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.981766 1177 raft_consensus.cc:740] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.981969 1177 consensus_queue.cc:260] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.982796 1183 tablet_bootstrap.cc:492] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:16.982949 1183 ts_tablet_manager.cc:1403] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251029 08:52:16.983740 1183 raft_consensus.cc:359] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.983927 1183 raft_consensus.cc:385] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.983978 1177 ts_tablet_manager.cc:1434] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:16.984000 1183 raft_consensus.cc:740] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.984261 1177 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.984268 1183 consensus_queue.cc:260] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.984758 1183 ts_tablet_manager.cc:1434] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:16.984897 747 tablet_service.cc:1505] Processing CreateTablet for tablet 39bd0c8760fb4c3093c67cf01af79cc7 (DEFAULT_TABLE table=default.Table_e5472287b7984f8f8a0fbd29cee00c13 [id=87fbc692c3274f5ba176618de97f15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.985081 1183 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:16.985287 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd0c8760fb4c3093c67cf01af79cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.985822 1177 tablet_bootstrap.cc:654] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.987633 1177 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:16.987731 1177 ts_tablet_manager.cc:1403] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251029 08:52:16.984897 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 (DEFAULT_TABLE table=default.Table_079d27f45f6a49e093d99a073678a387 [id=09890bc173754a139eec6c4eecbefddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:16.988339 1177 raft_consensus.cc:359] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.988438 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8970ca1f227849a1a99ac5d33e450e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:16.988478 1177 raft_consensus.cc:385] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:16.988521 1177 raft_consensus.cc:740] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:16.988698 1179 raft_consensus.cc:2243] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:16.988683 1177 consensus_queue.cc:260] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:16.989218 1177 ts_tablet_manager.cc:1434] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251029 08:52:16.989259 1179 pending_rounds.cc:70] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:16.989319 1179 pending_rounds.cc:77] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037629648289792 op_type: NO_OP noop_request { }
I20251029 08:52:16.989468 1177 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:16.989455 1179 raft_consensus.cc:2889] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:16.989540 1179 raft_consensus.cc:2272] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:16.991200 1177 tablet_bootstrap.cc:654] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:16.992781 1179 ts_tablet_manager.cc:1916] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:16.992825 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 116cc0d155e24653937953eea4f39ee4
I20251029 08:52:16.993420 589 catalog_manager.cc:6719] ReplaceTablet: tablet 116cc0d155e24653937953eea4f39ee4 deleted and replaced by tablet b61829fcd59d4ad4ab427270b7089d57
W20251029 08:52:16.995378 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b5e1e3ce83b14817b2440370cb2fcabd (table default.Table_3f1b48395900452b8b149c6aef010592 [id=fd07f1c36584477caf673f91f2a86f58], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b5e1e3ce83b14817b2440370cb2fcabd consensus state has no leader
I20251029 08:52:16.995786 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 116cc0d155e24653937953eea4f39ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet b61829fcd59d4ad4ab427270b7089d57) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:16.996002 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 116cc0d155e24653937953eea4f39ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet b61829fcd59d4ad4ab427270b7089d57) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:16.898391 2000 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 08:52:16.884545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40435 (local address 127.0.104.2:51343)
1029 08:52:16.898045 (+ 13500us) negotiation.cc:107] Waiting for socket to connect
1029 08:52:16.898149 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13354,"threads_started":1}
I20251029 08:52:17.001801 902 tablet_service.cc:1505] Processing CreateTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 (DEFAULT_TABLE table=default.Table_079d27f45f6a49e093d99a073678a387 [id=09890bc173754a139eec6c4eecbefddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.002185 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8970ca1f227849a1a99ac5d33e450e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.002889 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 431ab59e96274dde83ded7981b93e3d4 (DEFAULT_TABLE table=default.Table_d38046985bde419896eda670b4597795 [id=ad773f9a02ac482093113d2ed28c332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.003155 1183 tablet_bootstrap.cc:654] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.003258 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431ab59e96274dde83ded7981b93e3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.004117 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 116cc0d155e24653937953eea4f39ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet b61829fcd59d4ad4ab427270b7089d57) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.001801 896 tablet_service.cc:1505] Processing CreateTablet for tablet 39bd0c8760fb4c3093c67cf01af79cc7 (DEFAULT_TABLE table=default.Table_e5472287b7984f8f8a0fbd29cee00c13 [id=87fbc692c3274f5ba176618de97f15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.004539 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd0c8760fb4c3093c67cf01af79cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.004794 1178 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.004909 1178 ts_tablet_manager.cc:1403] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20251029 08:52:17.005625 1178 raft_consensus.cc:359] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.005781 1178 raft_consensus.cc:385] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.005827 1178 raft_consensus.cc:740] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.006058 1178 consensus_queue.cc:260] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.006686 1178 ts_tablet_manager.cc:1434] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.006940 1178 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.008641 1178 tablet_bootstrap.cc:654] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.010818 1179 ts_tablet_manager.cc:1929] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:17.010983 1179 log.cc:1199] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:17.011407 1179 ts_tablet_manager.cc:1950] T 757e34137e884d5ca5cd88fdd03244d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:17.014308 1974 ts_tablet_manager.cc:1929] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.015755 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ba1bcb3f6c134de2b322696501927a16 (table default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]) successfully deleted
I20251029 08:52:17.016846 764 tablet_service.cc:1505] Processing CreateTablet for tablet 431ab59e96274dde83ded7981b93e3d4 (DEFAULT_TABLE table=default.Table_d38046985bde419896eda670b4597795 [id=ad773f9a02ac482093113d2ed28c332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.017215 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431ab59e96274dde83ded7981b93e3d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:17.017661 861 consensus_peers.cc:597] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:17.018769 1974 log.cc:1199] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:17.019198 1974 ts_tablet_manager.cc:1950] T bc96ef1a94844a32923087c44a5bcfc8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.024124 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_180698a62af24093bee27beaedfc0c18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.024680 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_180698a62af24093bee27beaedfc0c18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.026757 763 tablet_service.cc:1505] Processing CreateTablet for tablet 20006242722c4134b53340632af61698 (DEFAULT_TABLE table=default.Table_6541220eef6e47c1861484bbf1a5f44c [id=e0e1007d3df44d48a2b844d083f14f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.027086 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20006242722c4134b53340632af61698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.027019 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 7d24187f389a45628d21370c5b29a537 (DEFAULT_TABLE table=default.Table_d9b9417a07e84ab38c3498a14cbbb03d [id=18e7b6df8fdf4cf3b05da59c548c7ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.027405 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d24187f389a45628d21370c5b29a537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.028285 762 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9cdcec8e494050a475defd0cfd119f (DEFAULT_TABLE table=default.Table_71c355ca7af84373b74ea04307f6a818 [id=b4f7c5be41214b80940cb984e6373535]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.028630 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9cdcec8e494050a475defd0cfd119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.028932 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 20006242722c4134b53340632af61698 (DEFAULT_TABLE table=default.Table_6541220eef6e47c1861484bbf1a5f44c [id=e0e1007d3df44d48a2b844d083f14f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.029331 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20006242722c4134b53340632af61698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.031136 764 tablet_service.cc:1505] Processing CreateTablet for tablet 7d24187f389a45628d21370c5b29a537 (DEFAULT_TABLE table=default.Table_d9b9417a07e84ab38c3498a14cbbb03d [id=18e7b6df8fdf4cf3b05da59c548c7ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.031173 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9cdcec8e494050a475defd0cfd119f (DEFAULT_TABLE table=default.Table_71c355ca7af84373b74ea04307f6a818 [id=b4f7c5be41214b80940cb984e6373535]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.031450 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d24187f389a45628d21370c5b29a537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.031544 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9cdcec8e494050a475defd0cfd119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.031865 1937 raft_consensus.cc:493] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.031992 1937 raft_consensus.cc:515] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.032634 1937 leader_election.cc:290] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.032748 1937 raft_consensus.cc:493] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.032840 1937 raft_consensus.cc:515] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.033329 1937 leader_election.cc:290] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.034077 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a9f0b579ebd4d2db813459611d7f9bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.034477 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14123bfd9c3a4a6aba8cacd92e0ac0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:17.034993 720 leader_election.cc:343] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.035180 720 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.036140 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 8052ae8345ad4f3f95714d7b9173d3d8 (DEFAULT_TABLE table=default.Table_c64ecb648f8c411fa562430e00d12375 [id=cc75992a109f4ce5a4dec49aeb0dffb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.036527 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8052ae8345ad4f3f95714d7b9173d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.036909 759 tablet_service.cc:1505] Processing CreateTablet for tablet 8052ae8345ad4f3f95714d7b9173d3d8 (DEFAULT_TABLE table=default.Table_c64ecb648f8c411fa562430e00d12375 [id=cc75992a109f4ce5a4dec49aeb0dffb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.037215 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8052ae8345ad4f3f95714d7b9173d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.037879 1183 tablet_bootstrap.cc:492] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.038028 1183 ts_tablet_manager.cc:1403] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.002s
I20251029 08:52:17.038870 1183 raft_consensus.cc:359] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.039045 1183 raft_consensus.cc:385] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.039139 1183 raft_consensus.cc:740] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.039361 1183 consensus_queue.cc:260] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.040118 1038 tablet_service.cc:1505] Processing CreateTablet for tablet f8a896408fd142b9b41d606738234ba4 (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.040247 761 tablet_service.cc:1505] Processing CreateTablet for tablet f8a896408fd142b9b41d606738234ba4 (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.040472 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a896408fd142b9b41d606738234ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.040542 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a896408fd142b9b41d606738234ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.043788 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 29bb35cd36464dbd96470cc78faa481f (DEFAULT_TABLE table=default.Table_79ee6c7183a04b5c93eb9a318008b975 [id=527d2bf488d94ef2bd3677020aa684b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.044133 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29bb35cd36464dbd96470cc78faa481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.044274 760 tablet_service.cc:1505] Processing CreateTablet for tablet 41622daeb1e0419fa88cc26c0405c2db (DEFAULT_TABLE table=default.Table_c253bd6532c8472295cf1d50dfc59d42 [id=0bb370d9e9e54c6c8b90b2b334f7a705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.044577 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41622daeb1e0419fa88cc26c0405c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.047647 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 41622daeb1e0419fa88cc26c0405c2db (DEFAULT_TABLE table=default.Table_c253bd6532c8472295cf1d50dfc59d42 [id=0bb370d9e9e54c6c8b90b2b334f7a705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.047963 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41622daeb1e0419fa88cc26c0405c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.043939 751 tablet_service.cc:1505] Processing CreateTablet for tablet 29bb35cd36464dbd96470cc78faa481f (DEFAULT_TABLE table=default.Table_79ee6c7183a04b5c93eb9a318008b975 [id=527d2bf488d94ef2bd3677020aa684b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.049075 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29bb35cd36464dbd96470cc78faa481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.040139 1183 ts_tablet_manager.cc:1434] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.049831 1178 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.049947 1178 ts_tablet_manager.cc:1403] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.003s
I20251029 08:52:17.050683 1178 raft_consensus.cc:359] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.050838 1178 raft_consensus.cc:385] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.050887 1178 raft_consensus.cc:740] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.051107 1178 consensus_queue.cc:260] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.052748 1178 ts_tablet_manager.cc:1434] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:17.053854 1178 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.053815 1050 tablet_service.cc:1505] Processing CreateTablet for tablet ab487db71c7d4355b136872e138c9c57 (DEFAULT_TABLE table=default.Table_37caed2a62614c98a11dab8ae8b51b2e [id=e353e8ccc01c4eefa5c7aa95a255e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.054162 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab487db71c7d4355b136872e138c9c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.054690 755 tablet_service.cc:1505] Processing CreateTablet for tablet ab487db71c7d4355b136872e138c9c57 (DEFAULT_TABLE table=default.Table_37caed2a62614c98a11dab8ae8b51b2e [id=e353e8ccc01c4eefa5c7aa95a255e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.055003 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab487db71c7d4355b136872e138c9c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.055438 1178 tablet_bootstrap.cc:654] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.056300 1044 tablet_service.cc:1505] Processing CreateTablet for tablet ff1b7181e52041288c45b053124bd844 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.056595 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b7181e52041288c45b053124bd844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.058470 752 tablet_service.cc:1505] Processing CreateTablet for tablet ff1b7181e52041288c45b053124bd844 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.058765 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b7181e52041288c45b053124bd844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.059962 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 0d08838d55e248ba99e00532bb82b65f (DEFAULT_TABLE table=default.Table_1c4199c5e69245c5a9e8cae1a4ed98da [id=7aa97e70d17b4fcb89af7f7028148a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.060312 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d08838d55e248ba99e00532bb82b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.060267 757 tablet_service.cc:1505] Processing CreateTablet for tablet 0d08838d55e248ba99e00532bb82b65f (DEFAULT_TABLE table=default.Table_1c4199c5e69245c5a9e8cae1a4ed98da [id=7aa97e70d17b4fcb89af7f7028148a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.060581 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d08838d55e248ba99e00532bb82b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.061374 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b60d8ff960f148d586e443888f754174" } new_table_name: "default.Table_c9c0266a3d25476cae75216163270a5f" modify_external_catalogs: true
I20251029 08:52:17.060325 1183 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.065729 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d7e9b2817baa4c87a5adad8a8905c6ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.066375 1183 tablet_bootstrap.cc:654] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.018797 888 tablet_service.cc:1505] Processing CreateTablet for tablet 431ab59e96274dde83ded7981b93e3d4 (DEFAULT_TABLE table=default.Table_d38046985bde419896eda670b4597795 [id=ad773f9a02ac482093113d2ed28c332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.025411 895 tablet_service.cc:1505] Processing CreateTablet for tablet 7d24187f389a45628d21370c5b29a537 (DEFAULT_TABLE table=default.Table_d9b9417a07e84ab38c3498a14cbbb03d [id=18e7b6df8fdf4cf3b05da59c548c7ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.026015 892 tablet_service.cc:1505] Processing CreateTablet for tablet 20006242722c4134b53340632af61698 (DEFAULT_TABLE table=default.Table_6541220eef6e47c1861484bbf1a5f44c [id=e0e1007d3df44d48a2b844d083f14f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.037051 1177 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:17.068313 1177 ts_tablet_manager.cc:1403] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.007s
I20251029 08:52:17.069036 1177 raft_consensus.cc:359] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.069216 1177 raft_consensus.cc:385] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.069279 1177 raft_consensus.cc:740] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.069484 1177 consensus_queue.cc:260] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.048728 1946 raft_consensus.cc:493] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.069833 1946 raft_consensus.cc:515] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.070421 1946 leader_election.cc:290] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.071849 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14b665b63a7341d699d499ffc8cfd9f7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:17.072342 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7e9b2817baa4c87a5adad8a8905c6ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:17.076164 861 leader_election.cc:343] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.076428 861 leader_election.cc:343] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 14b665b63a7341d699d499ffc8cfd9f7
I20251029 08:52:17.076530 861 leader_election.cc:304] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.076943 1946 raft_consensus.cc:2749] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.077924 1040 tablet_service.cc:1505] Processing CreateTablet for tablet b45f9177e6a74612872a43ddcba89972 (DEFAULT_TABLE table=default.Table_aa5dc97a2e214cc2bb4aeaeb28706945 [id=11e024554d33456181a076698530bc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.078125 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 82fdad2bc86541b4ae20690fae62fb42 (DEFAULT_TABLE table=default.Table_61da851751cb48c99d7c6dded2f21a43 [id=8bdbbb6aec044478a731e9980496fb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.078347 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45f9177e6a74612872a43ddcba89972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.078562 1183 tablet_bootstrap.cc:492] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.078675 1046 tablet_service.cc:1505] Processing CreateTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 (DEFAULT_TABLE table=default.Table_e66832af5e114b158d59c2326322b87c [id=2a98498adcc9497f83353476affdd6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.078763 1183 ts_tablet_manager.cc:1403] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251029 08:52:17.079046 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4ac3f024a74eb592a1de6fd8f57985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.079793 1183 raft_consensus.cc:359] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.080061 1183 raft_consensus.cc:385] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.080149 1183 raft_consensus.cc:740] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.080295 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 (DEFAULT_TABLE table=default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.080428 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fdad2bc86541b4ae20690fae62fb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.080641 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e35d40b730a4a7883c324a6cc93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.080479 1183 consensus_queue.cc:260] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.080291 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 5d292c29d2b34086a5976da251f9ed5e (DEFAULT_TABLE table=default.Table_6a0b95dd9d0341d697d8709cd5e42be3 [id=a777f7827a99404e81d969c3f2c2c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.081070 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d292c29d2b34086a5976da251f9ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.081276 1183 ts_tablet_manager.cc:1434] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 08:52:17.081532 1928 raft_consensus.cc:493] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.081687 1928 raft_consensus.cc:515] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.082535 1928 leader_election.cc:290] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.083112 1183 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.085335 1183 tablet_bootstrap.cc:654] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.087175 1178 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.087318 1178 ts_tablet_manager.cc:1403] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.009s
I20251029 08:52:17.088294 1178 raft_consensus.cc:359] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.088515 1178 raft_consensus.cc:385] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.088574 1178 raft_consensus.cc:740] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.088807 1178 consensus_queue.cc:260] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.089453 1178 ts_tablet_manager.cc:1434] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:17.089756 1178 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.091308 748 tablet_service.cc:1505] Processing CreateTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 (DEFAULT_TABLE table=default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.091411 1178 tablet_bootstrap.cc:654] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.091439 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.091637 782 raft_consensus.cc:2393] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:17.091693 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e35d40b730a4a7883c324a6cc93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.095149 750 tablet_service.cc:1505] Processing CreateTablet for tablet 82fdad2bc86541b4ae20690fae62fb42 (DEFAULT_TABLE table=default.Table_61da851751cb48c99d7c6dded2f21a43 [id=8bdbbb6aec044478a731e9980496fb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.095468 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fdad2bc86541b4ae20690fae62fb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.095465 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3c9cdcec8e494050a475defd0cfd119f (DEFAULT_TABLE table=default.Table_71c355ca7af84373b74ea04307f6a818 [id=b4f7c5be41214b80940cb984e6373535]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.095989 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8052ae8345ad4f3f95714d7b9173d3d8 (DEFAULT_TABLE table=default.Table_c64ecb648f8c411fa562430e00d12375 [id=cc75992a109f4ce5a4dec49aeb0dffb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.096489 899 tablet_service.cc:1505] Processing CreateTablet for tablet f8a896408fd142b9b41d606738234ba4 (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.096935 886 tablet_service.cc:1505] Processing CreateTablet for tablet 29bb35cd36464dbd96470cc78faa481f (DEFAULT_TABLE table=default.Table_79ee6c7183a04b5c93eb9a318008b975 [id=527d2bf488d94ef2bd3677020aa684b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.097421 897 tablet_service.cc:1505] Processing CreateTablet for tablet 41622daeb1e0419fa88cc26c0405c2db (DEFAULT_TABLE table=default.Table_c253bd6532c8472295cf1d50dfc59d42 [id=0bb370d9e9e54c6c8b90b2b334f7a705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.097862 891 tablet_service.cc:1505] Processing CreateTablet for tablet ab487db71c7d4355b136872e138c9c57 (DEFAULT_TABLE table=default.Table_37caed2a62614c98a11dab8ae8b51b2e [id=e353e8ccc01c4eefa5c7aa95a255e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.098274 898 tablet_service.cc:1505] Processing CreateTablet for tablet ff1b7181e52041288c45b053124bd844 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.098769 890 tablet_service.cc:1505] Processing CreateTablet for tablet 0d08838d55e248ba99e00532bb82b65f (DEFAULT_TABLE table=default.Table_1c4199c5e69245c5a9e8cae1a4ed98da [id=7aa97e70d17b4fcb89af7f7028148a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.099047 758 tablet_service.cc:1505] Processing CreateTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 (DEFAULT_TABLE table=default.Table_e66832af5e114b158d59c2326322b87c [id=2a98498adcc9497f83353476affdd6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.099092 1946 raft_consensus.cc:493] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.099211 1946 raft_consensus.cc:515] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.099349 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4ac3f024a74eb592a1de6fd8f57985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.099822 1946 leader_election.cc:290] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.100157 893 tablet_service.cc:1505] Processing CreateTablet for tablet b45f9177e6a74612872a43ddcba89972 (DEFAULT_TABLE table=default.Table_aa5dc97a2e214cc2bb4aeaeb28706945 [id=11e024554d33456181a076698530bc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.101262 900 tablet_service.cc:1505] Processing CreateTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 (DEFAULT_TABLE table=default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.101768 901 tablet_service.cc:1505] Processing CreateTablet for tablet 82fdad2bc86541b4ae20690fae62fb42 (DEFAULT_TABLE table=default.Table_61da851751cb48c99d7c6dded2f21a43 [id=8bdbbb6aec044478a731e9980496fb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.102281 887 tablet_service.cc:1505] Processing CreateTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 (DEFAULT_TABLE table=default.Table_e66832af5e114b158d59c2326322b87c [id=2a98498adcc9497f83353476affdd6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.091315 745 tablet_service.cc:1505] Processing CreateTablet for tablet b45f9177e6a74612872a43ddcba89972 (DEFAULT_TABLE table=default.Table_aa5dc97a2e214cc2bb4aeaeb28706945 [id=11e024554d33456181a076698530bc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.102703 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45f9177e6a74612872a43ddcba89972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.104709 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef0a7b31ced24388bc9d9964a42fa203" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:17.105119 861 leader_election.cc:343] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ef0a7b31ced24388bc9d9964a42fa203
W20251029 08:52:17.105437 861 leader_election.cc:343] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.105538 861 leader_election.cc:304] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.106621 1946 raft_consensus.cc:2749] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.107554 746 tablet_service.cc:1505] Processing CreateTablet for tablet 5d292c29d2b34086a5976da251f9ed5e (DEFAULT_TABLE table=default.Table_6a0b95dd9d0341d697d8709cd5e42be3 [id=a777f7827a99404e81d969c3f2c2c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.107635 1928 raft_consensus.cc:493] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.107869 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d292c29d2b34086a5976da251f9ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.107784 1928 raft_consensus.cc:515] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.108484 1928 leader_election.cc:290] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.111387 1178 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.111515 1178 ts_tablet_manager.cc:1403] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20251029 08:52:17.112316 1178 raft_consensus.cc:359] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.112463 1178 raft_consensus.cc:385] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.112509 1178 raft_consensus.cc:740] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.112709 1178 consensus_queue.cc:260] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:17.112838 998 leader_election.cc:343] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:17.114153 1178 ts_tablet_manager.cc:1434] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:17.114274 1042 tablet_service.cc:1505] Processing CreateTablet for tablet b8f8a87bf57e4b1e9d08336c544b472b (DEFAULT_TABLE table=default.Table_656e3c6b753e4109bc607af874503567 [id=adf2271f85d846aaa17081ed3bf14dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.114642 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f8a87bf57e4b1e9d08336c544b472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.115183 1178 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.115302 754 tablet_service.cc:1505] Processing CreateTablet for tablet b8f8a87bf57e4b1e9d08336c544b472b (DEFAULT_TABLE table=default.Table_656e3c6b753e4109bc607af874503567 [id=adf2271f85d846aaa17081ed3bf14dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.115608 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f8a87bf57e4b1e9d08336c544b472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.116840 1178 tablet_bootstrap.cc:654] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.118582 1183 tablet_bootstrap.cc:492] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.118716 1183 ts_tablet_manager.cc:1403] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20251029 08:52:17.119688 1183 raft_consensus.cc:359] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.119875 1183 raft_consensus.cc:385] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.119930 1183 raft_consensus.cc:740] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.120155 1183 consensus_queue.cc:260] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.120833 1183 ts_tablet_manager.cc:1434] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.121140 1183 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.121914 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 29bb35cd36464dbd96470cc78faa481f
I20251029 08:52:17.122571 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4d371c09f6534a4db843f2ee86ef54df" } new_table_name: "default.Table_ef3c7626a5fc4a63949adbb110e6fdf0" modify_external_catalogs: true
I20251029 08:52:17.122785 1183 tablet_bootstrap.cc:654] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.124425 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 29bb35cd36464dbd96470cc78faa481f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.125726 1179 tablet_replica.cc:333] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:17.125905 1179 raft_consensus.cc:2243] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.126034 1179 raft_consensus.cc:2272] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.126147 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 29bb35cd36464dbd96470cc78faa481f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.126852 1179 ts_tablet_manager.cc:1916] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.133525 753 tablet_service.cc:1505] Processing CreateTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 (DEFAULT_TABLE table=default.Table_78e22440874d4598ac910219a652f3ed [id=23c1f5e63fdf483fa6d87a774be74a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.133875 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6d85081f246788ba29bb4a9c9a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.135385 1179 ts_tablet_manager.cc:1929] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.135502 1179 log.cc:1199] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c9a1d5e1f68e4f0e8b78641bbdc98123
I20251029 08:52:17.135818 1179 ts_tablet_manager.cc:1950] T c9a1d5e1f68e4f0e8b78641bbdc98123 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:17.137924 1178 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.138075 1178 ts_tablet_manager.cc:1403] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251029 08:52:17.138808 1183 tablet_bootstrap.cc:492] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.138939 1183 ts_tablet_manager.cc:1403] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20251029 08:52:17.138870 1178 raft_consensus.cc:359] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.139017 1178 raft_consensus.cc:385] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.139070 1178 raft_consensus.cc:740] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.139190 749 tablet_service.cc:1505] Processing CreateTablet for tablet e8aa13bbe1df4df3b10837ce2f8d0c90 (DEFAULT_TABLE table=default.Table_f4b5630f70e44d8886ee111768d79161 [id=64bb100c10ab4d60bd6288cd5e12e477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.139261 1178 consensus_queue.cc:260] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.139542 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aa13bbe1df4df3b10837ce2f8d0c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.139606 1183 raft_consensus.cc:359] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.139721 1178 ts_tablet_manager.cc:1434] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.139750 1183 raft_consensus.cc:385] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.139797 1183 raft_consensus.cc:740] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.139986 1178 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.140010 1183 consensus_queue.cc:260] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.140586 1183 ts_tablet_manager.cc:1434] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.140851 1183 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.142225 1183 tablet_bootstrap.cc:654] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.143023 1979 raft_consensus.cc:493] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.143175 1979 raft_consensus.cc:515] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.143888 1979 leader_election.cc:290] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.145205 1178 tablet_bootstrap.cc:654] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.145859 1183 tablet_bootstrap.cc:492] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.145977 1183 ts_tablet_manager.cc:1403] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251029 08:52:17.146888 1183 raft_consensus.cc:359] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.147066 1183 raft_consensus.cc:385] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.147131 1183 raft_consensus.cc:740] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.147390 1183 consensus_queue.cc:260] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.148052 1183 ts_tablet_manager.cc:1434] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.148372 1183 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.150429 1183 tablet_bootstrap.cc:654] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.153079 1178 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.153251 1178 ts_tablet_manager.cc:1403] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251029 08:52:17.154225 1178 raft_consensus.cc:359] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.154443 1178 raft_consensus.cc:385] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.154515 1178 raft_consensus.cc:740] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.154814 1178 consensus_queue.cc:260] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.155371 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.155822 1183 tablet_bootstrap.cc:492] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.155973 1183 ts_tablet_manager.cc:1403] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251029 08:52:17.155979 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 757e34137e884d5ca5cd88fdd03244d8 (table default.Table_4a022119f31d4460af5cdd68baa8521b [id=9eab4fb7fff44c10854a84cbf994f667]) successfully deleted
I20251029 08:52:17.156409 1178 ts_tablet_manager.cc:1434] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:17.156479 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c9a1d5e1f68e4f0e8b78641bbdc98123 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
W20251029 08:52:17.156697 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 116cc0d155e24653937953eea4f39ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116cc0d155e24653937953eea4f39ee4 already in progress: creating tablet
I20251029 08:52:17.156752 1183 raft_consensus.cc:359] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.156909 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 29bb35cd36464dbd96470cc78faa481f because the tablet was not found. No further retry: Not found: Tablet not found: 29bb35cd36464dbd96470cc78faa481f
I20251029 08:52:17.156960 1183 raft_consensus.cc:385] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.157044 1183 raft_consensus.cc:740] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.157303 1183 consensus_queue.cc:260] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.157501 1178 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.157795 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7 (DEFAULT_TABLE table=default.Table_e733598233c746a3aeab693a18a55a0f [id=466181d2378143b892c0c3824ab814ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.157934 1183 ts_tablet_manager.cc:1434] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:17.158217 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.158277 1183 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.159114 1178 tablet_bootstrap.cc:654] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.159878 1050 tablet_service.cc:1505] Processing CreateTablet for tablet d87191d05d4145a796a8823f723bc499 (DEFAULT_TABLE table=default.Table_4876937bd3244d43b4154fdd5eb81de7 [id=8e09762a58b44257b5e4e8a6aef4f2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.160377 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87191d05d4145a796a8823f723bc499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.163086 1049 tablet_service.cc:1505] Processing CreateTablet for tablet e8aa13bbe1df4df3b10837ce2f8d0c90 (DEFAULT_TABLE table=default.Table_f4b5630f70e44d8886ee111768d79161 [id=64bb100c10ab4d60bd6288cd5e12e477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.163436 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aa13bbe1df4df3b10837ce2f8d0c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.168293 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 81c2c690e3364f3abb61bc01bbb2980e (DEFAULT_TABLE table=default.Table_505dbf6dc0c64448a4ef6f67807dff75 [id=3a1d11053256402bb6a631d9adae1656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.159858 1183 tablet_bootstrap.cc:654] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.168720 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c2c690e3364f3abb61bc01bbb2980e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:17.172451 720 leader_election.cc:343] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.173619 1044 tablet_service.cc:1505] Processing CreateTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 (DEFAULT_TABLE table=default.Table_78e22440874d4598ac910219a652f3ed [id=23c1f5e63fdf483fa6d87a774be74a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.173980 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6d85081f246788ba29bb4a9c9a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.174264 1183 tablet_bootstrap.cc:492] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.174439 1183 ts_tablet_manager.cc:1403] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20251029 08:52:17.175382 1183 raft_consensus.cc:359] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.175599 1183 raft_consensus.cc:385] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.175711 1183 raft_consensus.cc:740] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.175959 1183 consensus_queue.cc:260] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.176898 1183 ts_tablet_manager.cc:1434] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.176928 1178 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.177042 1178 ts_tablet_manager.cc:1403] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20251029 08:52:17.177186 1183 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.177805 1178 raft_consensus.cc:359] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.177879 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 9a603e0ef22e48f18263ece15b799c0c (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.177989 1178 raft_consensus.cc:385] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.178046 1178 raft_consensus.cc:740] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.178026 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 (DEFAULT_TABLE table=default.Table_aae83ce4ba7f4a1288306bf92745e84e [id=198adaf31b9145a38e51e0ece130029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.178301 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a603e0ef22e48f18263ece15b799c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.178246 1178 consensus_queue.cc:260] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.178700 1178 ts_tablet_manager.cc:1434] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.178740 1033 tablet_service.cc:1505] Processing CreateTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 (DEFAULT_TABLE table=default.Table_64a621b6339340e99357f629eeb43501 [id=b8364f5fdc644d40b8ba54a382fea4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.178937 1183 tablet_bootstrap.cc:654] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.178985 1178 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.179111 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e3f702f9a407abf4fe7c7be513c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.181316 1183 tablet_bootstrap.cc:492] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.181485 1183 ts_tablet_manager.cc:1403] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251029 08:52:17.181841 1178 tablet_bootstrap.cc:654] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.182262 1979 raft_consensus.cc:493] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.182291 1183 raft_consensus.cc:359] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.182389 1979 raft_consensus.cc:515] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.182490 1183 raft_consensus.cc:385] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.182575 1183 raft_consensus.cc:740] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.182864 1183 consensus_queue.cc:260] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.183197 1979 leader_election.cc:290] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.183243 1050 tablet_service.cc:1505] Processing CreateTablet for tablet f7a6ef03f18c4ee78f2f10f31d1d8058 (DEFAULT_TABLE table=default.Table_e67f5ce3a1ac4caab580b9e7727e6d4a [id=afc30e0fa2464e558fcc3a7961a470e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.183480 1183 ts_tablet_manager.cc:1434] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.183681 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a6ef03f18c4ee78f2f10f31d1d8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.183835 1183 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.185844 1183 tablet_bootstrap.cc:654] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.188639 1178 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.188818 1178 ts_tablet_manager.cc:1403] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20251029 08:52:17.188691 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 63e705a0b5264eb7a6988129edbf5970 (DEFAULT_TABLE table=default.Table_f713c7b9b580405c87b7fdc7df46acf4 [id=cd8c1f442cc84086be3e06a0dc5337e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.188959 1043 tablet_service.cc:1505] Processing CreateTablet for tablet d66eea455961438da21c085156a9733d (DEFAULT_TABLE table=default.Table_4d371c09f6534a4db843f2ee86ef54df [id=5874be82ad554795a8c4d234ceee4838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.189249 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.189450 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66eea455961438da21c085156a9733d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.189724 1178 raft_consensus.cc:359] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.189927 1178 raft_consensus.cc:385] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.189986 1178 raft_consensus.cc:740] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.190223 1979 raft_consensus.cc:493] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.190339 1979 raft_consensus.cc:515] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.190959 1979 leader_election.cc:290] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.191206 1183 tablet_bootstrap.cc:492] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.191370 1183 ts_tablet_manager.cc:1403] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20251029 08:52:17.191422 1178 consensus_queue.cc:260] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.192212 1183 raft_consensus.cc:359] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.192425 1183 raft_consensus.cc:385] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.192544 1183 raft_consensus.cc:740] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.192800 1183 consensus_queue.cc:260] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.192974 1178 ts_tablet_manager.cc:1434] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:17.193496 1183 ts_tablet_manager.cc:1434] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.193804 1183 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.194031 1178 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.195649 1183 tablet_bootstrap.cc:654] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.195708 1178 tablet_bootstrap.cc:654] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.199548 747 tablet_service.cc:1505] Processing CreateTablet for tablet 81c2c690e3364f3abb61bc01bbb2980e (DEFAULT_TABLE table=default.Table_505dbf6dc0c64448a4ef6f67807dff75 [id=3a1d11053256402bb6a631d9adae1656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.199476 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8991ea78321545be85d5c480841e4751" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.199606 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bda3e143699b41b6bba308c83bbb3e0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.197774 1183 tablet_bootstrap.cc:492] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.203245 1183 ts_tablet_manager.cc:1403] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20251029 08:52:17.203320 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fd4ac3f024a74eb592a1de6fd8f57985
I20251029 08:52:17.204102 1183 raft_consensus.cc:359] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.204152 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e7328a4d3185459186db060425ff7e13" } new_table_name: "default.Table_327cf682312b426891ded0b7e60f5242" modify_external_catalogs: true
I20251029 08:52:17.204320 1183 raft_consensus.cc:385] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.204403 1183 raft_consensus.cc:740] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.204686 1183 consensus_queue.cc:260] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.205250 1183 ts_tablet_manager.cc:1434] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.205778 1183 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.206403 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 98324d4ae1af45fa94ae438dc2a64aaa
I20251029 08:52:17.207013 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2f8ccb5f14eb48eb87e67d8d788ca18a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.207521 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f8ccb5f14eb48eb87e67d8d788ca18a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.207702 1183 tablet_bootstrap.cc:654] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.208674 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2f4c8c74a7b64d4ba5d9318bf6b23601
I20251029 08:52:17.209218 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_736d75bfbe144b3887b5d49fa7088845" } new_table_name: "default.Table_bb583028431740f4afd61a171a14298e" modify_external_catalogs: true
I20251029 08:52:17.210379 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6689cf0804f946c18ebc555a191549d4
I20251029 08:52:17.210672 1183 tablet_bootstrap.cc:492] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.210817 1183 ts_tablet_manager.cc:1403] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251029 08:52:17.210968 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e4e823c7272f44e1af831ef60277d505" } new_table_name: "default.Table_d2fa4bcfb5a240239a27ceb1bebf974c" modify_external_catalogs: true
I20251029 08:52:17.211680 1183 raft_consensus.cc:359] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.211872 1183 raft_consensus.cc:385] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.211961 1183 raft_consensus.cc:740] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.212236 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 7046bb1fcdd34d308aecddfdf0752911 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:17.212248 1183 consensus_queue.cc:260] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.212745 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_343f0a80fc4c4dceb7098e5f75a81389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.212952 1183 ts_tablet_manager.cc:1434] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251029 08:52:17.213186 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_343f0a80fc4c4dceb7098e5f75a81389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.213245 1183 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:17.214309 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 667c71a8dfd54c05ab56e845653c0c9a consensus state has no leader
I20251029 08:52:17.214636 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_124a72a977e04bf391ade81da1a92c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.215045 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_124a72a977e04bf391ade81da1a92c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.215061 1183 tablet_bootstrap.cc:654] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.216241 591 master_service.cc:954] ReplaceTablet: received request to replace tablet d306be1d51674c2abc506d248e05a9c7 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:17.217571 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_693b1227de8f476d97871a176deda38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.218014 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_693b1227de8f476d97871a176deda38f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.221794 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f323cb89e8a7460991eb7cfc5da8b754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.222194 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f323cb89e8a7460991eb7cfc5da8b754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.225138 1183 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.225317 1183 ts_tablet_manager.cc:1403] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20251029 08:52:17.226207 1183 raft_consensus.cc:359] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.226437 1183 raft_consensus.cc:385] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.226513 1183 raft_consensus.cc:740] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.226759 1183 consensus_queue.cc:260] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.227468 1183 ts_tablet_manager.cc:1434] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.227798 1183 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.229807 1183 tablet_bootstrap.cc:654] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.232288 1183 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.232453 1183 ts_tablet_manager.cc:1403] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251029 08:52:17.233381 1183 raft_consensus.cc:359] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.233605 1183 raft_consensus.cc:385] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.233703 1183 raft_consensus.cc:740] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.233949 1183 consensus_queue.cc:260] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.234714 1183 ts_tablet_manager.cc:1434] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.235040 1183 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.235882 1946 raft_consensus.cc:493] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.236071 1946 raft_consensus.cc:515] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.237041 1946 leader_election.cc:290] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.238790 1183 tablet_bootstrap.cc:654] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.251833 1183 tablet_bootstrap.cc:492] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.252028 1183 ts_tablet_manager.cc:1403] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20251029 08:52:17.252920 1183 raft_consensus.cc:359] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.253121 1183 raft_consensus.cc:385] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.253209 1183 raft_consensus.cc:740] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.253424 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.253510 1183 consensus_queue.cc:260] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.253672 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.253871 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.254019 1183 ts_tablet_manager.cc:1434] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:17.254339 1183 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.254462 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.260147 1751 raft_consensus.cc:493] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.260295 1751 raft_consensus.cc:515] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.260979 1751 leader_election.cc:290] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.261760 1946 raft_consensus.cc:493] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.261929 1946 raft_consensus.cc:515] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.262599 1946 leader_election.cc:290] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:17.263401 1183 tablet_bootstrap.cc:654] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.263708 904 tablet_service.cc:1505] Processing CreateTablet for tablet 5d292c29d2b34086a5976da251f9ed5e (DEFAULT_TABLE table=default.Table_6a0b95dd9d0341d697d8709cd5e42be3 [id=a777f7827a99404e81d969c3f2c2c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.264624 905 tablet_service.cc:1505] Processing CreateTablet for tablet b8f8a87bf57e4b1e9d08336c544b472b (DEFAULT_TABLE table=default.Table_656e3c6b753e4109bc607af874503567 [id=adf2271f85d846aaa17081ed3bf14dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.265115 1946 raft_consensus.cc:493] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.265268 1946 raft_consensus.cc:515] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.265374 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 29bb35cd36464dbd96470cc78faa481f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.265792 1946 leader_election.cc:290] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.265913 903 tablet_service.cc:1505] Processing CreateTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 (DEFAULT_TABLE table=default.Table_78e22440874d4598ac910219a652f3ed [id=23c1f5e63fdf483fa6d87a774be74a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.269707 925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:17.270699 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 because the tablet was not found. No further retry: Not found: Tablet not found: 2f4c8c74a7b64d4ba5d9318bf6b23601
W20251029 08:52:17.270993 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6689cf0804f946c18ebc555a191549d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6689cf0804f946c18ebc555a191549d4 already in progress: creating tablet
W20251029 08:52:17.271185 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 98324d4ae1af45fa94ae438dc2a64aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98324d4ae1af45fa94ae438dc2a64aaa already in progress: creating tablet
W20251029 08:52:17.271394 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fd4ac3f024a74eb592a1de6fd8f57985 because the tablet was not found. No further retry: Not found: Tablet not found: fd4ac3f024a74eb592a1de6fd8f57985
I20251029 08:52:17.269709 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c5008d9df9419bb4b6d678c6dc27a6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.272152 921 raft_consensus.cc:2468] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.272377 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a9f0b579ebd4d2db813459611d7f9bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.272552 923 raft_consensus.cc:2468] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:17.272763 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14123bfd9c3a4a6aba8cacd92e0ac0bc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.274734 1183 tablet_bootstrap.cc:492] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.274874 1183 ts_tablet_manager.cc:1403] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251029 08:52:17.275663 1183 raft_consensus.cc:359] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.275825 1183 raft_consensus.cc:385] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.275878 1183 raft_consensus.cc:740] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.276126 1183 consensus_queue.cc:260] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.276266 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8991ea78321545be85d5c480841e4751" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.276542 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bda3e143699b41b6bba308c83bbb3e0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.276644 1183 ts_tablet_manager.cc:1434] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.276793 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc57cd5a8d3c42ba90f2011084e8b4bd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.276908 1183 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.277278 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea6d85081f246788ba29bb4a9c9a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.278785 1183 tablet_bootstrap.cc:654] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.280369 1974 tablet_replica.cc:333] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:17.280642 1974 raft_consensus.cc:2243] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.280822 1974 raft_consensus.cc:2272] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.281486 1974 ts_tablet_manager.cc:1916] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.285887 896 tablet_service.cc:1505] Processing CreateTablet for tablet f7a6ef03f18c4ee78f2f10f31d1d8058 (DEFAULT_TABLE table=default.Table_e67f5ce3a1ac4caab580b9e7727e6d4a [id=afc30e0fa2464e558fcc3a7961a470e9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:17.289163 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85b300f17a704f7dba1aa3724fcc2de7 consensus state has no leader
I20251029 08:52:17.289259 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6bfec7dc813f42d588da6bd24fe2f3b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.289863 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bfec7dc813f42d588da6bd24fe2f3b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:17.292369 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d66eea455961438da21c085156a9733d consensus state has no leader
W20251029 08:52:17.292821 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a69beee4032f447eaa694d237cb0e4e3 consensus state has no leader
W20251029 08:52:17.293226 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a9f0b579ebd4d2db813459611d7f9bf consensus state has no leader
I20251029 08:52:17.293929 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d306be1d51674c2abc506d248e05a9c7
I20251029 08:52:17.294473 591 catalog_manager.cc:6719] ReplaceTablet: tablet d306be1d51674c2abc506d248e05a9c7 deleted and replaced by tablet a08630cbc401467a9dde1ce1e198cef7
I20251029 08:52:17.299575 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet d306be1d51674c2abc506d248e05a9c7 with delete_type TABLET_DATA_DELETED (replaced by tablet a08630cbc401467a9dde1ce1e198cef7) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:17.313040 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d306be1d51674c2abc506d248e05a9c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d306be1d51674c2abc506d248e05a9c7 already in progress: creating tablet
I20251029 08:52:17.319625 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:17.319984 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]). Expected version 2 got 0
I20251029 08:52:17.329975 996 leader_election.cc:304] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.330662 1751 raft_consensus.cc:2804] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.330765 1751 raft_consensus.cc:493] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.330843 1751 raft_consensus.cc:3060] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.334424 1751 raft_consensus.cc:515] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.335889 1751 leader_election.cc:290] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.337674 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d24187f389a45628d21370c5b29a537. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.350754 996 leader_election.cc:304] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.351804 1751 raft_consensus.cc:3060] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.353195 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3fb78e6a50342688581081eb505f8ee" } modify_external_catalogs: true
I20251029 08:52:17.353538 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3fb78e6a50342688581081eb505f8ee" } modify_external_catalogs: true
I20251029 08:52:17.355551 1751 raft_consensus.cc:2749] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.410574 589 master_service.cc:954] ReplaceTablet: received request to replace tablet e4190dc9e09b4480a79774d1e0b5bc2c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:17.202801 748 tablet_service.cc:1505] Processing CreateTablet for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7 (DEFAULT_TABLE table=default.Table_e733598233c746a3aeab693a18a55a0f [id=466181d2378143b892c0c3824ab814ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.209667 1178 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.420305 1178 ts_tablet_manager.cc:1403] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.000s
I20251029 08:52:17.210798 758 tablet_service.cc:1505] Processing CreateTablet for tablet d87191d05d4145a796a8823f723bc499 (DEFAULT_TABLE table=default.Table_4876937bd3244d43b4154fdd5eb81de7 [id=8e09762a58b44257b5e4e8a6aef4f2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.213742 745 tablet_service.cc:1505] Processing CreateTablet for tablet f7a6ef03f18c4ee78f2f10f31d1d8058 (DEFAULT_TABLE table=default.Table_e67f5ce3a1ac4caab580b9e7727e6d4a [id=afc30e0fa2464e558fcc3a7961a470e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.216842 746 tablet_service.cc:1505] Processing CreateTablet for tablet d66eea455961438da21c085156a9733d (DEFAULT_TABLE table=default.Table_4d371c09f6534a4db843f2ee86ef54df [id=5874be82ad554795a8c4d234ceee4838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.219637 754 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 (DEFAULT_TABLE table=default.Table_aae83ce4ba7f4a1288306bf92745e84e [id=198adaf31b9145a38e51e0ece130029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.420953 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87191d05d4145a796a8823f723bc499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.226737 749 tablet_service.cc:1505] Processing CreateTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 (DEFAULT_TABLE table=default.Table_64a621b6339340e99357f629eeb43501 [id=b8364f5fdc644d40b8ba54a382fea4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.421257 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e3f702f9a407abf4fe7c7be513c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.228772 1979 raft_consensus.cc:493] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.231658 753 tablet_service.cc:1505] Processing CreateTablet for tablet 9a603e0ef22e48f18263ece15b799c0c (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.422048 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a603e0ef22e48f18263ece15b799c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.248032 750 tablet_service.cc:1552] Processing DeleteTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.248080 1937 raft_consensus.cc:493] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.248181 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.422832 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66eea455961438da21c085156a9733d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.422865 1751 raft_consensus.cc:493] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.423139 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a6ef03f18c4ee78f2f10f31d1d8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.423126 1751 raft_consensus.cc:515] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.251098 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.254218 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.424459 1751 leader_election.cc:290] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.261565 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.424772 782 raft_consensus.cc:2468] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:17.270181 718 leader_election.cc:343] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.425064 718 leader_election.cc:304] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:17.229064 720 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.425275 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c2c690e3364f3abb61bc01bbb2980e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:17.425257 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 196191us
I20251029 08:52:17.380764 2040 raft_consensus.cc:493] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.404758 2041 raft_consensus.cc:493] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.425753 1183 tablet_bootstrap.cc:492] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.425907 1183 ts_tablet_manager.cc:1403] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.149s user 0.053s sys 0.077s
I20251029 08:52:17.427054 1183 raft_consensus.cc:359] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.427258 1183 raft_consensus.cc:385] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.427330 1183 raft_consensus.cc:740] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.428370 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.428412 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f7b89ced286b4a04a28022917b1d0235
I20251029 08:52:17.428434 1183 consensus_queue.cc:260] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.429880 1183 ts_tablet_manager.cc:1434] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:17.430932 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet f7b89ced286b4a04a28022917b1d0235 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.431347 1183 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:17.433027 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f7b89ced286b4a04a28022917b1d0235 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b89ced286b4a04a28022917b1d0235 already in progress: creating tablet
I20251029 08:52:17.433902 2040 raft_consensus.cc:515] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.434633 2040 leader_election.cc:290] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.434772 2040 raft_consensus.cc:493] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.434887 2040 raft_consensus.cc:515] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.435413 2040 leader_election.cc:290] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:17.435539 2040 raft_consensus.cc:493] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.435635 2040 raft_consensus.cc:515] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.436165 2040 leader_election.cc:290] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:17.436614 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 166430us
I20251029 08:52:17.437691 718 leader_election.cc:304] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
W20251029 08:52:17.438030 718 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.438122 718 leader_election.cc:304] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.438365 1979 raft_consensus.cc:515] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.438832 1979 leader_election.cc:290] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.438948 1979 raft_consensus.cc:2749] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.439070 1979 raft_consensus.cc:2804] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.422451 1937 raft_consensus.cc:515] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.439580 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b7181e52041288c45b053124bd844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.440788 998 leader_election.cc:304] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.439601 1937 leader_election.cc:290] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:17.442253 1937 raft_consensus.cc:2749] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.440146 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:17.445835 718 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.445947 718 leader_election.cc:304] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.446187 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ff59f466b5804429bc09b24c36220b1f" } modify_external_catalogs: true
I20251029 08:52:17.446379 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ff59f466b5804429bc09b24c36220b1f" } modify_external_catalogs: true
I20251029 08:52:17.447314 1183 tablet_bootstrap.cc:654] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.447999 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9cd744dfdd12453386d05dbdb493b993" } modify_external_catalogs: true
I20251029 08:52:17.448220 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9cd744dfdd12453386d05dbdb493b993" } modify_external_catalogs: true
I20251029 08:52:17.439122 1979 raft_consensus.cc:493] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.448462 1979 raft_consensus.cc:3060] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:17.448935 718 leader_election.cc:343] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.449704 998 leader_election.cc:343] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c5008d9df9419bb4b6d678c6dc27a6
W20251029 08:52:17.449918 998 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:17.449839 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_76bc2e4759364fe488fa32a2afa440a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.450383 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76bc2e4759364fe488fa32a2afa440a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.450692 1937 raft_consensus.cc:2749] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.450846 1937 raft_consensus.cc:493] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.452144 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_439c96c50e724a6db3abaac6c5540a36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.446256 2041 raft_consensus.cc:515] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:17.452698 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_439c96c50e724a6db3abaac6c5540a36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.452847 2041 leader_election.cc:290] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:17.454366 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5dc644387108496995ffad8f788f3c24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.454849 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc644387108496995ffad8f788f3c24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.456231 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b0593e0dafd4dfda7b1b2b9fb545383" } new_table_name: "default.Table_062158d11a604fa6b931f3c1501da486" modify_external_catalogs: true
I20251029 08:52:17.457619 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a97711da23ff4ab79b7f1810e23031c6" } modify_external_catalogs: true
I20251029 08:52:17.457794 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a97711da23ff4ab79b7f1810e23031c6" } modify_external_catalogs: true
I20251029 08:52:17.458477 1178 raft_consensus.cc:359] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.458647 1178 raft_consensus.cc:385] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.458704 1178 raft_consensus.cc:740] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.458921 1178 consensus_queue.cc:260] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.459544 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b843daec43c24b55a07bbfa6d6fc5a5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.460045 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b843daec43c24b55a07bbfa6d6fc5a5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.460279 1178 ts_tablet_manager.cc:1434] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251029 08:52:17.461253 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64f9de5287d54b348a406a42a7c379ab" } modify_external_catalogs: true
I20251029 08:52:17.461421 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64f9de5287d54b348a406a42a7c379ab" } modify_external_catalogs: true
I20251029 08:52:17.464810 1928 raft_consensus.cc:2804] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.464924 1928 raft_consensus.cc:493] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.465016 1928 raft_consensus.cc:3060] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.467612 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e705a0b5264eb7a6988129edbf5970. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.468125 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bc96ef1a94844a32923087c44a5bcfc8 (table default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]) successfully deleted
I20251029 08:52:17.439467 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.470292 1928 raft_consensus.cc:515] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.472335 1928 leader_election.cc:290] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.472484 1928 raft_consensus.cc:493] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.472582 1928 raft_consensus.cc:515] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.473060 1928 leader_election.cc:290] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.479084 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431ab59e96274dde83ded7981b93e3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.479645 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f8a87bf57e4b1e9d08336c544b472b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.480178 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41622daeb1e0419fa88cc26c0405c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.482511 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d292c29d2b34086a5976da251f9ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.484179 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fdad2bc86541b4ae20690fae62fb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.488147 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20006242722c4134b53340632af61698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.488656 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45f9177e6a74612872a43ddcba89972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.492151 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8052ae8345ad4f3f95714d7b9173d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.492380 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d08838d55e248ba99e00532bb82b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.496155 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a896408fd142b9b41d606738234ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.496381 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a6ef03f18c4ee78f2f10f31d1d8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.496291 902 tablet_service.cc:1505] Processing CreateTablet for tablet e8aa13bbe1df4df3b10837ce2f8d0c90 (DEFAULT_TABLE table=default.Table_f4b5630f70e44d8886ee111768d79161 [id=64bb100c10ab4d60bd6288cd5e12e477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.496867 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8aa13bbe1df4df3b10837ce2f8d0c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.499802 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29bb35cd36464dbd96470cc78faa481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.500298 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4ac3f024a74eb592a1de6fd8f57985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.470453 1937 raft_consensus.cc:515] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.504339 752 tablet_service.cc:1552] Processing DeleteTablet for tablet d306be1d51674c2abc506d248e05a9c7 with delete_type TABLET_DATA_DELETED (replaced by tablet a08630cbc401467a9dde1ce1e198cef7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.504565 757 tablet_service.cc:1552] Processing DeleteTablet for tablet f7b89ced286b4a04a28022917b1d0235 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.504781 1937 leader_election.cc:290] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.506500 1974 ts_tablet_manager.cc:1929] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.506722 1974 log.cc:1199] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:17.507186 1974 ts_tablet_manager.cc:1950] T a8c5008d9df9419bb4b6d678c6dc27a6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.509585 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:17.509799 780 raft_consensus.cc:3060] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:17.512990 998 leader_election.cc:343] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3295bb0b47714080a5c3f34049acd487
I20251029 08:52:17.513973 780 raft_consensus.cc:2468] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.514326 1937 raft_consensus.cc:493] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.514551 998 leader_election.cc:304] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.514505 1937 raft_consensus.cc:515] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.514863 1751 raft_consensus.cc:2804] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.514978 1751 raft_consensus.cc:697] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.515228 1751 consensus_queue.cc:237] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.515327 1937 leader_election.cc:290] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.516341 1928 raft_consensus.cc:493] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.516498 1928 raft_consensus.cc:515] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.517746 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.517953 780 raft_consensus.cc:2468] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.518904 998 leader_election.cc:304] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.519996 1937 raft_consensus.cc:493] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.520150 1937 raft_consensus.cc:515] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.520840 1937 leader_election.cc:290] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.508576 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c9cdcec8e494050a475defd0cfd119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.522188 1974 tablet_replica.cc:333] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:17.522480 1974 raft_consensus.cc:2243] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.517859 903 tablet_service.cc:1505] Processing CreateTablet for tablet 81c2c690e3364f3abb61bc01bbb2980e (DEFAULT_TABLE table=default.Table_505dbf6dc0c64448a4ef6f67807dff75 [id=3a1d11053256402bb6a631d9adae1656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.525270 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c2c690e3364f3abb61bc01bbb2980e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.525523 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab487db71c7d4355b136872e138c9c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.528249 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e35d40b730a4a7883c324a6cc93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.535619 1928 leader_election.cc:290] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.536535 1928 raft_consensus.cc:2804] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.537109 1928 raft_consensus.cc:493] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.537555 1928 raft_consensus.cc:3060] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.538518 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e4190dc9e09b4480a79774d1e0b5bc2c
I20251029 08:52:17.539249 589 catalog_manager.cc:6719] ReplaceTablet: tablet e4190dc9e09b4480a79774d1e0b5bc2c deleted and replaced by tablet 47f58be6d7f5492395ca6192f7ce0930
I20251029 08:52:17.539475 757 tablet_service.cc:1552] Processing DeleteTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c with delete_type TABLET_DATA_DELETED (replaced by tablet 47f58be6d7f5492395ca6192f7ce0930) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.521378 898 tablet_service.cc:1505] Processing CreateTablet for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7 (DEFAULT_TABLE table=default.Table_e733598233c746a3aeab693a18a55a0f [id=466181d2378143b892c0c3824ab814ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.540272 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14795a3b422e4c74ac6ef1cdb5e6f9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.540553 1946 raft_consensus.cc:493] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.540714 1946 raft_consensus.cc:515] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.541357 1946 leader_election.cc:290] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.541297 1928 raft_consensus.cc:515] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.542268 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:17.542466 780 raft_consensus.cc:3060] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.542461 1946 raft_consensus.cc:493] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.542599 1946 raft_consensus.cc:515] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.543164 1946 leader_election.cc:290] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.544176 1974 raft_consensus.cc:2272] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.544744 1974 ts_tablet_manager.cc:1916] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.545184 780 raft_consensus.cc:2468] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.545732 998 leader_election.cc:304] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.549012 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "084a659932ec4996a6d6b160d67efdd0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.549058 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30183814f5e4f2ab9d71fe493eb801c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.549750 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.550462 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351047c725f8488abbd074bd40790b48" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.552441 888 tablet_service.cc:1505] Processing CreateTablet for tablet d87191d05d4145a796a8823f723bc499 (DEFAULT_TABLE table=default.Table_4876937bd3244d43b4154fdd5eb81de7 [id=8e09762a58b44257b5e4e8a6aef4f2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.552908 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87191d05d4145a796a8823f723bc499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.555539 893 tablet_service.cc:1505] Processing CreateTablet for tablet d66eea455961438da21c085156a9733d (DEFAULT_TABLE table=default.Table_4d371c09f6534a4db843f2ee86ef54df [id=5874be82ad554795a8c4d234ceee4838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.555889 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66eea455961438da21c085156a9733d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.559043 891 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 (DEFAULT_TABLE table=default.Table_aae83ce4ba7f4a1288306bf92745e84e [id=198adaf31b9145a38e51e0ece130029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.559365 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.560971 902 tablet_service.cc:1505] Processing CreateTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 (DEFAULT_TABLE table=default.Table_64a621b6339340e99357f629eeb43501 [id=b8364f5fdc644d40b8ba54a382fea4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.561339 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e3f702f9a407abf4fe7c7be513c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.563488 1979 raft_consensus.cc:515] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.564502 1979 leader_election.cc:290] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.567795 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afa2bb1075c4418eb10f4493f5e73c1b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.568476 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7da7e08b74f4a9aacedada42e839a11" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.568954 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.569367 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc9c221eaa9a4721ba25761206a13645" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.569778 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.570254 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a9f0b579ebd4d2db813459611d7f9bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.570418 920 raft_consensus.cc:3060] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.571625 1178 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.573503 920 raft_consensus.cc:2468] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:17.574894 757 tablet_service.cc:1505] Processing CreateTablet for tablet 63e705a0b5264eb7a6988129edbf5970 (DEFAULT_TABLE table=default.Table_f713c7b9b580405c87b7fdc7df46acf4 [id=cd8c1f442cc84086be3e06a0dc5337e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.575399 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e705a0b5264eb7a6988129edbf5970. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:17.583978 1998 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 08:52:16.880444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40435 (local address 127.0.104.3:59479)
1029 08:52:17.582618 (+702174us) negotiation.cc:107] Waiting for socket to connect
1029 08:52:17.583745 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":701993,"threads_started":1}
W20251029 08:52:17.584777 997 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.010s sys 0.001s
W20251029 08:52:17.584872 997 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.704s user 0.010s sys 0.001s
W20251029 08:52:17.585650 583 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.000s sys 0.011s
W20251029 08:52:17.585764 583 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.711s user 0.000s sys 0.011s
W20251029 08:52:17.585865 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 710833us
I20251029 08:52:17.588435 1928 leader_election.cc:290] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.588639 1928 raft_consensus.cc:2804] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.588761 1928 raft_consensus.cc:697] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.590163 1928 consensus_queue.cc:237] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.597790 2044 raft_consensus.cc:493] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.597970 2044 raft_consensus.cc:515] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.598685 996 leader_election.cc:343] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.599138 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c with delete_type TABLET_DATA_DELETED (replaced by tablet 47f58be6d7f5492395ca6192f7ce0930) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.605579 2047 raft_consensus.cc:493] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.605859 2047 raft_consensus.cc:515] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.606673 2047 leader_election.cc:290] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.606892 2047 raft_consensus.cc:493] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.607051 2047 raft_consensus.cc:515] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.607712 2047 leader_election.cc:290] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:17.610343 583 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b -> Peer f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435): Couldn't send request to peer f718bd0f00004f20a89d028a2fa8e6b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:17.612269 902 tablet_service.cc:1505] Processing CreateTablet for tablet 9a603e0ef22e48f18263ece15b799c0c (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.612797 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a603e0ef22e48f18263ece15b799c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.621657 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 400c4991955b47e799df66c05caee6e3
I20251029 08:52:17.628968 903 tablet_service.cc:1552] Processing DeleteTablet for tablet fd4ac3f024a74eb592a1de6fd8f57985 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.629231 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 98324d4ae1af45fa94ae438dc2a64aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631109 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631278 888 tablet_service.cc:1552] Processing DeleteTablet for tablet d306be1d51674c2abc506d248e05a9c7 with delete_type TABLET_DATA_DELETED (replaced by tablet a08630cbc401467a9dde1ce1e198cef7) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631376 888 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631465 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631551 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631641 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631721 888 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631803 888 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.631886 888 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.635262 1183 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.635443 1183 ts_tablet_manager.cc:1403] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.204s user 0.034s sys 0.046s
I20251029 08:52:17.636320 1183 raft_consensus.cc:359] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.636534 1183 raft_consensus.cc:385] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.636588 1183 raft_consensus.cc:740] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.636766 1183 consensus_queue.cc:260] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.637027 2047 raft_consensus.cc:493] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.637038 1705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:17.637118 2047 raft_consensus.cc:515] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.637225 1705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
I20251029 08:52:17.638115 2047 leader_election.cc:290] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.639948 1183 ts_tablet_manager.cc:1434] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:17.641278 1183 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.645465 2044 leader_election.cc:290] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.646608 1183 tablet_bootstrap.cc:654] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:17.646966 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e4190dc9e09b4480a79774d1e0b5bc2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4190dc9e09b4480a79774d1e0b5bc2c already in progress: creating tablet
I20251029 08:52:17.651945 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e4190dc9e09b4480a79774d1e0b5bc2c with delete_type TABLET_DATA_DELETED (replaced by tablet 47f58be6d7f5492395ca6192f7ce0930) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.652585 896 tablet_service.cc:1505] Processing CreateTablet for tablet 63e705a0b5264eb7a6988129edbf5970 (DEFAULT_TABLE table=default.Table_f713c7b9b580405c87b7fdc7df46acf4 [id=cd8c1f442cc84086be3e06a0dc5337e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:17.653036 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e705a0b5264eb7a6988129edbf5970. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:17.656121 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.661342 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:17.662743 718 leader_election.cc:343] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.662994 718 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663151 718 leader_election.cc:343] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663332 718 leader_election.cc:343] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663486 718 leader_election.cc:343] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663625 718 leader_election.cc:343] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663755 718 leader_election.cc:343] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.663884 718 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.664008 718 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.664207 718 leader_election.cc:304] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:17.664642 1979 raft_consensus.cc:2804] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.664783 1979 raft_consensus.cc:697] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:17.669623 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 400c4991955b47e799df66c05caee6e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.671696 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 400c4991955b47e799df66c05caee6e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.672816 2044 raft_consensus.cc:493] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.673015 2044 raft_consensus.cc:515] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.674314 1974 ts_tablet_manager.cc:1929] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.674441 1974 log.cc:1199] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/14b665b63a7341d699d499ffc8cfd9f7
I20251029 08:52:17.674790 1974 ts_tablet_manager.cc:1950] T 14b665b63a7341d699d499ffc8cfd9f7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.665077 1979 consensus_queue.cc:237] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.677342 1751 raft_consensus.cc:493] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.677500 1751 raft_consensus.cc:515] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.679522 895 tablet_service.cc:1552] Processing DeleteTablet for tablet f7b89ced286b4a04a28022917b1d0235 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.682276 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 400c4991955b47e799df66c05caee6e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.684245 1751 leader_election.cc:290] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.685133 1974 tablet_replica.cc:333] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:17.685336 1974 raft_consensus.cc:2243] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.685492 1974 raft_consensus.cc:2272] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.691068 1176 ts_tablet_manager.cc:1929] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.691286 1176 log.cc:1199] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/07906300f1d54d4e8fde5165ee14c44f
I20251029 08:52:17.691696 1176 ts_tablet_manager.cc:1950] T 07906300f1d54d4e8fde5165ee14c44f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:17.699587 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a8c5008d9df9419bb4b6d678c6dc27a6 (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:17.702214 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 14b665b63a7341d699d499ffc8cfd9f7 (table default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]) successfully deleted
W20251029 08:52:17.704792 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 400c4991955b47e799df66c05caee6e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400c4991955b47e799df66c05caee6e3 already in progress: creating tablet
I20251029 08:52:17.706076 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a31847928db44bd4b1ebd50c5130065f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.706269 924 raft_consensus.cc:2468] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.714299 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d896959a4c684e1b8b68a36125e751b9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.715363 1974 ts_tablet_manager.cc:1916] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.716930 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.719162 2050 raft_consensus.cc:493] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.719362 2050 raft_consensus.cc:515] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.720129 2050 leader_election.cc:290] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.720816 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff4e625abf14fbd94a8102dfa5fc993" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.720870 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.721354 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2773cae9814df79ee60ea8b0fe4e9d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.721500 919 raft_consensus.cc:2468] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:17.722096 998 leader_election.cc:343] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: f33c0bbf104046ccb86ff313f08da22a
I20251029 08:52:17.722239 2038 raft_consensus.cc:493] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:17.722384 998 leader_election.cc:343] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0ff4e625abf14fbd94a8102dfa5fc993
I20251029 08:52:17.722424 2038 raft_consensus.cc:515] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.722654 998 leader_election.cc:343] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e2773cae9814df79ee60ea8b0fe4e9d
W20251029 08:52:17.722859 998 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: abc307b14aff4c5e937a5151aab1dfb9
I20251029 08:52:17.723073 2038 leader_election.cc:290] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.723757 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f33c0bbf104046ccb86ff313f08da22a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.723970 920 raft_consensus.cc:2468] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.724792 1178 tablet_bootstrap.cc:654] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.725723 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d896959a4c684e1b8b68a36125e751b9
W20251029 08:52:17.728204 998 leader_election.cc:343] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:17.728773 2050 raft_consensus.cc:493] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.729133 2050 raft_consensus.cc:515] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.730615 2050 leader_election.cc:290] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.735466 902 tablet_service.cc:1552] Processing DeleteTablet for tablet d896959a4c684e1b8b68a36125e751b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.721531 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.737051 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 689) took 1128 ms. Trace:
I20251029 08:52:17.737484 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 084a659932ec4996a6d6b160d67efdd0 (table default.Table_c36b90fc430a4a89ba87469be66cb2a5 [id=d88876b1c5514d78892db64b612182ba]). Expected version 1 got 0
I20251029 08:52:17.726392 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.743213 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:17.743611 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet bda3e143699b41b6bba308c83bbb3e0d (table default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]). Expected version 1 got 0
I20251029 08:52:17.743600 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdcc9a441c6a4d82a58262f4fdb3f00d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.743888 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5fa181849474a559701fdb8c18d398b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.744077 919 raft_consensus.cc:2468] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.744140 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet afa2bb1075c4418eb10f4493f5e73c1b (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]). Expected version 1 got 0
I20251029 08:52:17.747934 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]). Expected version 2 got 0
W20251029 08:52:17.750659 1177 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.004s sys 0.021s
W20251029 08:52:17.750767 1177 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.004s sys 0.021s
I20251029 08:52:17.739126 747 tablet_service.cc:1552] Processing DeleteTablet for tablet d896959a4c684e1b8b68a36125e751b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.739934 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.750898 1176 rpcz_store.cc:276] 1029 08:52:16.608740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.608790 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:16.846846 (+238056us) ts_tablet_manager.cc:1151] Deleting tablet 07906300f1d54d4e8fde5165ee14c44f
1029 08:52:16.846859 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:17.636922 (+790063us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.690944 (+ 54022us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.737021 (+ 46077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5368,"tablet-delete.queue_time_us":237050}
I20251029 08:52:17.751072 780 raft_consensus.cc:2468] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.751267 1176 tablet_replica.cc:333] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:17.740299 718 leader_election.cc:343] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.751466 1176 raft_consensus.cc:2243] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.751623 1176 raft_consensus.cc:2272] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.752068 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 07906300f1d54d4e8fde5165ee14c44f (table default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]) successfully deleted
I20251029 08:52:17.752136 1177 ts_tablet_manager.cc:1434] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.684s user 0.004s sys 0.021s
I20251029 08:52:17.752506 996 leader_election.cc:304] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:17.752985 996 leader_election.cc:343] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:17.753095 996 leader_election.cc:304] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:17.753464 996 leader_election.cc:343] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.753665 1928 raft_consensus.cc:2804] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.753790 1928 raft_consensus.cc:493] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.753922 1928 raft_consensus.cc:3060] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.749686 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.755489 2044 leader_election.cc:290] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:17.752215 1177 ts_tablet_manager.cc:1467] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Tablet startup took 683ms
I20251029 08:52:17.759382 1176 ts_tablet_manager.cc:1916] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.761301 1928 raft_consensus.cc:515] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.763042 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.763446 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.763700 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.754216 920 raft_consensus.cc:2468] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:17.766943 757 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.767189 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.767457 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.772130 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.772439 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee8117e04f5f40e98ea755dc461067dd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.772608 920 raft_consensus.cc:2468] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:17.774828 720 leader_election.cc:343] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.775210 720 leader_election.cc:304] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.776139 1928 leader_election.cc:290] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.776469 2042 raft_consensus.cc:2749] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.764237 996 leader_election.cc:343] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 3295bb0b47714080a5c3f34049acd487
I20251029 08:52:17.778553 996 leader_election.cc:304] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.753717 2050 raft_consensus.cc:2749] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.779060 1946 raft_consensus.cc:493] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.774444 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d896959a4c684e1b8b68a36125e751b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.779192 1946 raft_consensus.cc:515] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.779526 1179 tablet_replica.cc:333] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:17.779691 1179 raft_consensus.cc:2243] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:17.779692 1946 leader_election.cc:290] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.779882 1179 pending_rounds.cc:70] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:17.779914 1179 pending_rounds.cc:77] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037632064438272 op_type: NO_OP noop_request { }
I20251029 08:52:17.779985 1179 raft_consensus.cc:2889] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:17.780076 1179 raft_consensus.cc:2272] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.780385 2042 raft_consensus.cc:493] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.780520 2042 raft_consensus.cc:515] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.781018 1179 ts_tablet_manager.cc:1916] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.781103 2042 leader_election.cc:290] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.781198 2042 raft_consensus.cc:493] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.781284 2042 raft_consensus.cc:515] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.781769 2042 leader_election.cc:290] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.782186 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f40aafe7dc94851a67abd84d80d7d7b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.785106 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_703d8f64e5cd49e58f0354ed41369097" } new_table_name: "default.Table_068eb03057e140249e1b348fd34910ce" modify_external_catalogs: true
I20251029 08:52:17.786475 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_04ec7a02b4e44860b339d5e95468a084" } new_table_name: "default.Table_35d49c0bbc6d402a8b18885c966ad1e3" modify_external_catalogs: true
I20251029 08:52:17.788059 1179 ts_tablet_manager.cc:1929] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:17.788214 1179 log.cc:1199] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d896959a4c684e1b8b68a36125e751b9
W20251029 08:52:17.788745 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c32679cbb1884f09addb0f7c26ac941e consensus state has no leader
I20251029 08:52:17.789639 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0bd490da0b0e47fc965343c123a27b22
I20251029 08:52:17.790488 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.790514 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b2ca7c25488a477fbe498883e704971c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.791047 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2ca7c25488a477fbe498883e704971c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.795662 1179 ts_tablet_manager.cc:1950] T d896959a4c684e1b8b68a36125e751b9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:17.795619 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea13e9ea2c11479ea86e3fd7f2a9e59c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.796124 1974 ts_tablet_manager.cc:1929] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.796228 1974 log.cc:1199] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/0ff4e625abf14fbd94a8102dfa5fc993
W20251029 08:52:17.796247 718 leader_election.cc:343] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.796370 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.796474 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_327cf682312b426891ded0b7e60f5242" } new_table_name: "default.Table_34c608523fd44399b9c71a365266edd6" modify_external_catalogs: true
I20251029 08:52:17.795120 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d18c461ffec64795a25d91d924ffa054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.775065 998 leader_election.cc:343] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e5fa181849474a559701fdb8c18d398b
I20251029 08:52:17.762157 2044 raft_consensus.cc:493] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.796983 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3421bab54d2445d09e0be9c5e15ac6da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.796967 2044 raft_consensus.cc:515] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.797564 2044 leader_election.cc:290] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:17.758903 1177 ts_tablet_manager.cc:1469] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Trace:
1029 08:52:06.677582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:06.677638 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:06.678153 (+ 515us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 08:52:06.678174 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1029 08:52:06.683143 (+ 4969us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:06.687051 (+ 3908us) tablet_replica.cc:206] Creating consensus instance
1029 08:52:06.687246 (+ 195us) inbound_call.cc:173] Queueing success response
1029 08:52:16.989382 (+10302136us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1029 08:52:17.068403 (+ 79021us) ts_tablet_manager.cc:1435] Starting tablet replica
1029 08:52:17.068560 (+ 157us) tablet_replica.cc:261] Starting instrumentation
1029 08:52:17.068676 (+ 116us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":888,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":43,"tablet-open.queue_time_us":10302202,"thread_start_us":681160,"threads_started":1}
I20251029 08:52:17.799633 1177 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:17.800707 1974 ts_tablet_manager.cc:1950] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.776665 1183 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.801018 1183 ts_tablet_manager.cc:1403] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.003s
I20251029 08:52:17.801060 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 702) took 1080 ms. Trace:
I20251029 08:52:17.801120 1974 rpcz_store.cc:276] 1029 08:52:16.720673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.720936 (+ 263us) service_pool.cc:225] Handling call
1029 08:52:17.685093 (+964157us) ts_tablet_manager.cc:1151] Deleting tablet 0ff4e625abf14fbd94a8102dfa5fc993
1029 08:52:17.685099 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:17.766158 (+ 81059us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.796004 (+ 29846us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.801047 (+ 5043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":964039}
I20251029 08:52:17.801425 1974 tablet_replica.cc:333] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:17.801582 1974 raft_consensus.cc:2243] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.801700 1974 raft_consensus.cc:2272] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.801744 1183 raft_consensus.cc:359] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.801908 1183 raft_consensus.cc:385] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.801970 1183 raft_consensus.cc:740] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.802163 1183 consensus_queue.cc:260] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.802698 1183 ts_tablet_manager.cc:1434] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:17.803094 1974 ts_tablet_manager.cc:1916] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.803182 1979 raft_consensus.cc:493] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.803294 1979 raft_consensus.cc:515] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.802826 718 leader_election.cc:343] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.803858 1979 leader_election.cc:290] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.804245 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae561a995c94fadb288b6395c55a1bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.804318 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
W20251029 08:52:17.804659 718 leader_election.cc:343] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.804714 1183 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.804679 591 catalog_manager.cc:5649] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:17.806399 1183 tablet_bootstrap.cc:654] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.807476 1178 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.807613 1178 ts_tablet_manager.cc:1403] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.256s user 0.012s sys 0.001s
I20251029 08:52:17.808353 1178 raft_consensus.cc:359] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.808538 1178 raft_consensus.cc:385] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.808611 1178 raft_consensus.cc:740] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.808903 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:17.808858 1178 consensus_queue.cc:260] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.809303 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:17.809368 1178 ts_tablet_manager.cc:1434] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251029 08:52:17.779050 996 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:17.811812 996 leader_election.cc:343] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: Tablet not RUNNING: STOPPED
I20251029 08:52:17.811967 996 leader_election.cc:304] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.812402 996 leader_election.cc:304] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.812778 996 leader_election.cc:304] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.812820 1178 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.813360 1751 raft_consensus.cc:2804] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.778820 1928 raft_consensus.cc:2749] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.813438 1751 raft_consensus.cc:493] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.813511 1751 raft_consensus.cc:3060] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.813603 1928 raft_consensus.cc:2804] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.813678 1928 raft_consensus.cc:493] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.813735 1928 raft_consensus.cc:3060] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.815670 1928 raft_consensus.cc:515] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.813047 2044 raft_consensus.cc:2749] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.816210 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d18c461ffec64795a25d91d924ffa054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.816335 1751 raft_consensus.cc:515] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.816828 1751 leader_election.cc:290] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:17.817023 861 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.000s sys 0.022s
W20251029 08:52:17.817097 861 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.000s sys 0.026s
I20251029 08:52:17.817973 1178 tablet_bootstrap.cc:654] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.818099 594 catalog_manager.cc:5649] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:17.819681 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.819845 781 raft_consensus.cc:2468] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:17.820147 1177 tablet_bootstrap.cc:654] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.820533 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]). Expected version 2 got 0
I20251029 08:52:17.821235 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2067fe9013134f5b945ace2150cd86d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.821485 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:17.821619 781 raft_consensus.cc:2468] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:17.821746 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2067fe9013134f5b945ace2150cd86d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.822768 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:17.818538 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 0ff4e625abf14fbd94a8102dfa5fc993 (table default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]) successfully deleted
W20251029 08:52:17.813445 996 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.825407 996 leader_election.cc:304] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.825435 1974 ts_tablet_manager.cc:1929] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.825526 1974 log.cc:1199] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:17.825778 1974 ts_tablet_manager.cc:1950] T ee8117e04f5f40e98ea755dc461067dd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.825763 996 leader_election.cc:304] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.825968 1751 raft_consensus.cc:2749] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.826043 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 698) took 1106 ms. Trace:
W20251029 08:52:17.826066 996 leader_election.cc:343] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: abc307b14aff4c5e937a5151aab1dfb9
I20251029 08:52:17.826114 1751 raft_consensus.cc:2804] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.826185 1751 raft_consensus.cc:493] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.826174 996 leader_election.cc:304] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.826104 1974 rpcz_store.cc:276] 1029 08:52:16.719077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.721178 (+ 2101us) service_pool.cc:225] Handling call
1029 08:52:17.801383 (+1080205us) ts_tablet_manager.cc:1151] Deleting tablet ee8117e04f5f40e98ea755dc461067dd
1029 08:52:17.801391 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:17.806073 (+ 4682us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.825339 (+ 19266us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.826029 (+ 690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079998}
I20251029 08:52:17.826268 1751 raft_consensus.cc:3060] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.826378 2044 raft_consensus.cc:2749] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.827389 1928 leader_election.cc:290] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.810483 998 leader_election.cc:304] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.828794 1070 raft_consensus.cc:3060] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.828778 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ee8117e04f5f40e98ea755dc461067dd (table default.Table_d6dc6ba680874a51afe94c07adb83e2b [id=5730455f82a94d8698ac62979d87d9d6]) successfully deleted
I20251029 08:52:17.832607 1974 tablet_replica.cc:333] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:17.832530 1751 raft_consensus.cc:515] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.833045 1751 leader_election.cc:290] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.833266 1751 raft_consensus.cc:2804] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.833403 1751 raft_consensus.cc:493] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.833503 1751 raft_consensus.cc:3060] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.835294 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bbc4345cabbc476485b6a040a076ddaf
W20251029 08:52:17.836280 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9cabbbac8254a949750244e1d6991eb has no consensus state
I20251029 08:52:17.836534 1974 raft_consensus.cc:2243] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.837109 1751 raft_consensus.cc:515] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.837742 1751 leader_election.cc:290] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.840539 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff4e625abf14fbd94a8102dfa5fc993" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.840721 1071 raft_consensus.cc:2468] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:17.842283 1177 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:17.842294 996 leader_election.cc:304] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.842413 1177 ts_tablet_manager.cc:1403] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.001s
I20251029 08:52:17.843068 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a31847928db44bd4b1ebd50c5130065f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.843264 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d896959a4c684e1b8b68a36125e751b9 (table default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]) successfully deleted
I20251029 08:52:17.843225 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215cd666c71a4179a19e298868a88e96" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.843422 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2773cae9814df79ee60ea8b0fe4e9d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.843560 924 raft_consensus.cc:3060] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.843591 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f33c0bbf104046ccb86ff313f08da22a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.843791 922 raft_consensus.cc:3060] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.843241 920 raft_consensus.cc:3060] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:17.844368 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3421bab54d2445d09e0be9c5e15ac6da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.846258 924 raft_consensus.cc:2468] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.846319 920 raft_consensus.cc:2468] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.846640 922 raft_consensus.cc:2468] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.846732 748 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:17.847074 1751 raft_consensus.cc:2804] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.847227 1751 raft_consensus.cc:493] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.847349 1751 raft_consensus.cc:3060] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.848018 1946 consensus_queue.cc:1048] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:17.849160 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2773cae9814df79ee60ea8b0fe4e9d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.849460 1071 raft_consensus.cc:2393] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
W20251029 08:52:17.850380 861 leader_election.cc:343] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e2773cae9814df79ee60ea8b0fe4e9d
W20251029 08:52:17.850587 861 leader_election.cc:343] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0ff4e625abf14fbd94a8102dfa5fc993
W20251029 08:52:17.850780 861 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:17.851112 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a20f26d0f7874ccb960ffe4d27e854e1" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.850356 1177 raft_consensus.cc:359] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.851600 1177 raft_consensus.cc:385] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.851670 1177 raft_consensus.cc:740] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.851898 1177 consensus_queue.cc:260] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.852455 1178 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.852648 1178 ts_tablet_manager.cc:1403] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20251029 08:52:17.853443 1177 ts_tablet_manager.cc:1434] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251029 08:52:17.853400 1178 raft_consensus.cc:359] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.853679 1178 raft_consensus.cc:385] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.853767 1178 raft_consensus.cc:740] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.853921 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.854108 1071 raft_consensus.cc:2393] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:17.854066 1178 consensus_queue.cc:260] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.855571 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a7a673ba4fd4abcbfc3493c38dcc257" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.855628 1178 ts_tablet_manager.cc:1434] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:17.855722 1071 raft_consensus.cc:2468] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:17.855947 893 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.856097 1177 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:17.856726 861 leader_election.cc:304] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:17.856949 1178 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.857509 1177 tablet_bootstrap.cc:654] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.858217 1946 raft_consensus.cc:2804] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.858300 1946 raft_consensus.cc:493] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.858376 1946 raft_consensus.cc:3060] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.860980 1178 tablet_bootstrap.cc:654] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.861140 1946 raft_consensus.cc:515] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.861837 1946 leader_election.cc:290] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.862073 1974 raft_consensus.cc:2272] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.862568 1974 ts_tablet_manager.cc:1916] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.863626 2046 consensus_queue.cc:1048] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:17.865105 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.865254 1071 raft_consensus.cc:2468] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:17.866755 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
/tmp/dist-test-taskLKoUQR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--webserver_interface=127.0.104.62
--webserver_port=37745
--builtin_ntp_servers=127.0.104.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 08:52:17.867743 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5fa181849474a559701fdb8c18d398b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.867877 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:17.868167 924 raft_consensus.cc:2393] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:17.868259 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.867889 922 raft_consensus.cc:3060] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.868676 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "084a659932ec4996a6d6b160d67efdd0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.869132 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30183814f5e4f2ab9d71fe493eb801c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.869360 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.869676 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351047c725f8488abbd074bd40790b48" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.870018 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afa2bb1075c4418eb10f4493f5e73c1b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.870321 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7da7e08b74f4a9aacedada42e839a11" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.871043 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.871377 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc9c221eaa9a4721ba25761206a13645" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.871479 922 raft_consensus.cc:2468] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.871702 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.872102 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a9f0b579ebd4d2db813459611d7f9bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:17.872443 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.872635 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:17.872816 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f40aafe7dc94851a67abd84d80d7d7b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.873188 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea13e9ea2c11479ea86e3fd7f2a9e59c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:17.874475 998 leader_election.cc:343] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:17.874724 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 4200db822deb4ace936f561c230ac1fb from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:17.874889 998 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 215cd666c71a4179a19e298868a88e96
W20251029 08:52:17.875113 998 leader_election.cc:343] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: f33c0bbf104046ccb86ff313f08da22a
W20251029 08:52:17.875329 998 leader_election.cc:343] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e2773cae9814df79ee60ea8b0fe4e9d
W20251029 08:52:17.875540 998 leader_election.cc:343] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e5fa181849474a559701fdb8c18d398b
I20251029 08:52:17.875486 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_856c731e03944711af6efcd38024791e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.863623 861 leader_election.cc:343] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5a7a673ba4fd4abcbfc3493c38dcc257
W20251029 08:52:17.876106 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_856c731e03944711af6efcd38024791e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.876308 861 leader_election.cc:304] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:17.876559 2046 raft_consensus.cc:2804] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.876632 2046 raft_consensus.cc:493] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.876699 2046 raft_consensus.cc:3060] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.878567 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 63dfcd944c1f487b8911ae383770d930 (table default.Table_35d49c0bbc6d402a8b18885c966ad1e3 [id=acafbaf717604c2e87a1b9e6263dcf48]). Expected version 2 got 0
I20251029 08:52:17.879251 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 449ffc678b874871a672c3059235de6a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:17.879316 2046 raft_consensus.cc:515] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.879807 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c36b90fc430a4a89ba87469be66cb2a5" } modify_external_catalogs: true
I20251029 08:52:17.879873 2046 leader_election.cc:290] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.879941 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c36b90fc430a4a89ba87469be66cb2a5" } modify_external_catalogs: true
I20251029 08:52:17.880414 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c2a36b6e0bc74f5caaa6177fc6011231" } new_table_name: "default.Table_3fc2abebc9c249d6b0e46ab667bd9d21" modify_external_catalogs: true
I20251029 08:52:17.880722 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5d8159a04f9f454c8d3d5b4472097608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:17.881235 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d8159a04f9f454c8d3d5b4472097608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:17.882957 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c68d96e4128e496097a5c5a4740db7ac" } modify_external_catalogs: true
I20251029 08:52:17.883132 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c68d96e4128e496097a5c5a4740db7ac" } modify_external_catalogs: true
I20251029 08:52:17.886022 1751 raft_consensus.cc:515] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.887626 1751 leader_election.cc:290] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:17.874130 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:17.887915 782 raft_consensus.cc:3060] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:17.889218 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0bd490da0b0e47fc965343c123a27b22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bd490da0b0e47fc965343c123a27b22 already in progress: creating tablet
W20251029 08:52:17.889598 720 leader_election.cc:343] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.889698 720 leader_election.cc:304] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.889902 2042 raft_consensus.cc:2749] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.890360 720 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.890448 720 leader_election.cc:304] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.890655 2042 raft_consensus.cc:2749] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.890723 782 raft_consensus.cc:2468] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:17.874769 1979 raft_consensus.cc:493] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.890908 1979 raft_consensus.cc:515] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.886987 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]). Expected version 2 got 0
I20251029 08:52:17.891650 1979 leader_election.cc:290] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:17.883356 1176 ts_tablet_manager.cc:1929] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.892437 1176 log.cc:1199] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/449ffc678b874871a672c3059235de6a
I20251029 08:52:17.892865 1176 ts_tablet_manager.cc:1950] T 449ffc678b874871a672c3059235de6a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:17.893707 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 685) took 1286 ms. Trace:
W20251029 08:52:17.894250 720 leader_election.cc:343] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.894443 720 leader_election.cc:304] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.894749 1979 raft_consensus.cc:2749] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.895128 1176 rpcz_store.cc:276] 1029 08:52:16.607109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.608181 (+ 1072us) service_pool.cc:225] Handling call
1029 08:52:17.751218 (+1143037us) ts_tablet_manager.cc:1151] Deleting tablet 449ffc678b874871a672c3059235de6a
1029 08:52:17.751228 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:17.762674 (+ 11446us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.883169 (+120495us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.893687 (+ 10518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140173}
I20251029 08:52:17.895475 1177 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:17.895614 1177 ts_tablet_manager.cc:1403] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20251029 08:52:17.888931 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae561a995c94fadb288b6395c55a1bf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.896334 1177 raft_consensus.cc:359] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.896528 1177 raft_consensus.cc:385] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:17.896502 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bbc4345cabbc476485b6a040a076ddaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc4345cabbc476485b6a040a076ddaf already in progress: creating tablet
I20251029 08:52:17.896592 1177 raft_consensus.cc:740] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.893960 998 leader_election.cc:304] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:17.896809 1177 consensus_queue.cc:260] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.897194 998 leader_election.cc:343] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ee8117e04f5f40e98ea755dc461067dd
W20251029 08:52:17.895648 720 leader_election.cc:343] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.897559 720 leader_election.cc:304] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.897806 1979 raft_consensus.cc:2749] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.898206 720 leader_election.cc:343] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.898375 720 leader_election.cc:304] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.898655 1979 raft_consensus.cc:2749] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.899080 720 leader_election.cc:343] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.899241 720 leader_election.cc:304] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.899458 1979 raft_consensus.cc:2749] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.900264 720 leader_election.cc:343] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.900408 720 leader_election.cc:304] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.900491 1177 ts_tablet_manager.cc:1434] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:17.900784 1979 raft_consensus.cc:2749] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.901175 720 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.901324 720 leader_election.cc:304] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.901573 1979 raft_consensus.cc:2749] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.901942 720 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.902078 720 leader_election.cc:304] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.902290 1979 raft_consensus.cc:2749] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.902650 720 leader_election.cc:343] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.902794 1177 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:17.902846 748 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:17.902985 720 leader_election.cc:343] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.903110 720 leader_election.cc:304] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.903362 1979 raft_consensus.cc:2749] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.903771 720 leader_election.cc:343] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.903911 720 leader_election.cc:304] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.904168 1979 raft_consensus.cc:2749] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.904467 1177 tablet_bootstrap.cc:654] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.904706 1176 tablet_replica.cc:333] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:17.897410 1751 raft_consensus.cc:2804] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.905285 1751 raft_consensus.cc:697] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.905341 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 449ffc678b874871a672c3059235de6a (table default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]) successfully deleted
I20251029 08:52:17.905629 1751 consensus_queue.cc:237] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:17.906533 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85b300f17a704f7dba1aa3724fcc2de7 consensus state has no leader
I20251029 08:52:17.907829 1183 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:17.907981 1183 ts_tablet_manager.cc:1403] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.002s
I20251029 08:52:17.908460 861 leader_election.cc:304] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:17.908656 861 raft_consensus.cc:2686] T 0ff4e625abf14fbd94a8102dfa5fc993 P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:17.908962 1183 raft_consensus.cc:359] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.909245 1183 raft_consensus.cc:385] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.909349 1183 raft_consensus.cc:740] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.909677 861 leader_election.cc:304] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.909638 1183 consensus_queue.cc:260] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.909886 2046 raft_consensus.cc:2749] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:17.910307 861 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.910485 861 leader_election.cc:304] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.910506 589 master_service.cc:954] ReplaceTablet: received request to replace tablet a2b7853b85d24a04b8b11caaa4fe8c4d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:17.910574 1183 ts_tablet_manager.cc:1434] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:17.910764 2046 raft_consensus.cc:2749] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.911109 1183 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:17.911365 1176 raft_consensus.cc:2243] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:17.911695 1176 raft_consensus.cc:2272] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:17.912205 1178 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.912402 1178 ts_tablet_manager.cc:1403] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251029 08:52:17.912459 1176 ts_tablet_manager.cc:1916] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:17.913236 1178 raft_consensus.cc:359] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.913522 1178 raft_consensus.cc:385] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.913647 1178 raft_consensus.cc:740] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.913925 1178 consensus_queue.cc:260] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:17.914144 2046 consensus_queue.cc:1048] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:17.914693 861 leader_election.cc:304] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.914907 2046 raft_consensus.cc:2804] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:17.914927 1178 ts_tablet_manager.cc:1434] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:17.914990 2046 raft_consensus.cc:493] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:17.915063 2046 raft_consensus.cc:3060] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:17.904815 720 leader_election.cc:343] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.915530 1178 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.917155 1178 tablet_bootstrap.cc:654] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.917677 2046 raft_consensus.cc:515] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.918217 2046 leader_election.cc:290] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.918643 2046 consensus_queue.cc:1048] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:17.919193 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.915364 720 leader_election.cc:304] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.919649 1979 raft_consensus.cc:2749] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.920351 781 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:17.920626 781 raft_consensus.cc:2393] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:17.922042 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_836882de6c804a5c908824f31f0aa328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:17.922580 1183 tablet_bootstrap.cc:654] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.923029 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2db77250e3ef489e8ab7dbcfcbb54385" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:17.924099 2046 consensus_queue.cc:1048] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:17.925132 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.926960 1974 ts_tablet_manager.cc:1929] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:17.927067 1974 log.cc:1199] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/215cd666c71a4179a19e298868a88e96
I20251029 08:52:17.927385 1974 ts_tablet_manager.cc:1950] T 215cd666c71a4179a19e298868a88e96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:17.927685 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 705) took 1201 ms. Trace:
I20251029 08:52:17.927773 1974 rpcz_store.cc:276] 1029 08:52:16.725873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.725931 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:17.832554 (+1106623us) ts_tablet_manager.cc:1151] Deleting tablet 215cd666c71a4179a19e298868a88e96
1029 08:52:17.832567 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:17.923735 (+ 91168us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.926881 (+ 3146us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:17.927671 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25372,"tablet-delete.queue_time_us":1106480}
I20251029 08:52:17.928367 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:17.928570 1071 raft_consensus.cc:3060] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.931985 1071 raft_consensus.cc:2468] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:17.932616 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a7a673ba4fd4abcbfc3493c38dcc257" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:17.932834 1070 raft_consensus.cc:3060] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.935483 1070 raft_consensus.cc:2468] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:17.939908 996 leader_election.cc:343] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:17.940193 996 leader_election.cc:304] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:17.940634 996 leader_election.cc:304] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.940917 1928 raft_consensus.cc:2749] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:17.941107 1928 raft_consensus.cc:2804] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.941267 1928 raft_consensus.cc:697] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.941538 1928 consensus_queue.cc:237] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.956542 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898a4f58306b453e9abca2a73082bb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:17.956745 783 raft_consensus.cc:3060] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:17.959568 783 raft_consensus.cc:2468] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:17.960680 996 leader_election.cc:304] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.961247 996 leader_election.cc:304] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.961619 2050 raft_consensus.cc:2804] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.961840 2044 raft_consensus.cc:2804] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.961930 2044 raft_consensus.cc:697] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.962209 2044 consensus_queue.cc:237] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:17.963835 2050 raft_consensus.cc:697] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.964181 2050 consensus_queue.cc:237] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.984280 1178 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:17.984527 1178 ts_tablet_manager.cc:1403] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20251029 08:52:17.985759 1178 raft_consensus.cc:359] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.986069 1178 raft_consensus.cc:385] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:17.986181 1178 raft_consensus.cc:740] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:17.986477 1178 consensus_queue.cc:260] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:17.987658 1178 ts_tablet_manager.cc:1434] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:17.987989 1178 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:17.992815 1178 tablet_bootstrap.cc:654] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:17.997429 996 leader_election.cc:304] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:17.997892 1751 raft_consensus.cc:2804] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:17.998351 1751 raft_consensus.cc:697] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:17.998826 1751 consensus_queue.cc:237] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.015213 1176 ts_tablet_manager.cc:1929] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:18.017894 996 leader_election.cc:343] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:18.018149 996 leader_election.cc:304] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.018782 2047 raft_consensus.cc:2749] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.018930 2047 raft_consensus.cc:493] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.019018 2047 raft_consensus.cc:515] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.019582 2047 leader_election.cc:290] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:18.020720 998 leader_election.cc:343] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:18.025115 1751 raft_consensus.cc:493] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.025327 1751 raft_consensus.cc:515] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.026086 1751 leader_election.cc:290] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:18.027009 998 leader_election.cc:343] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 072cf27b0743488c8166e03b5e61c545
I20251029 08:52:18.031239 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a7b31ced24388bc9d9964a42fa203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.031342 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 14b665b63a7341d699d499ffc8cfd9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.031471 1179 tablet_replica.cc:333] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:18.031651 1179 raft_consensus.cc:2243] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.031823 1179 raft_consensus.cc:2272] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.032040 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet ba1bcb3f6c134de2b322696501927a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.032411 1179 ts_tablet_manager.cc:1916] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.032435 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 3295bb0b47714080a5c3f34049acd487 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:05 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.032817 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet a31847928db44bd4b1ebd50c5130065f with delete_type TABLET_DATA_DELETED (replaced by tablet b5e1e3ce83b14817b2440370cb2fcabd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.033190 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d896959a4c684e1b8b68a36125e751b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.033890 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff4e625abf14fbd94a8102dfa5fc993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.034219 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e2773cae9814df79ee60ea8b0fe4e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 9429f6abb50c4f66b2e1b565f303962d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.034580 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet e5fa181849474a559701fdb8c18d398b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:17.941735 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcaa3fa779824b38887ae001623ba67b consensus state has no leader
I20251029 08:52:18.035769 590 master_service.cc:954] ReplaceTablet: received request to replace tablet d5bb0fd4c62d47a2b10719ebd55851c0 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:18.036499 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b778f7a011e14d90a2f86ad390f9834f" } new_table_name: "default.Table_b2f1e53945774829a74b8125ee64a77a" modify_external_catalogs: true
I20251029 08:52:17.945175 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4200db822deb4ace936f561c230ac1fb
I20251029 08:52:18.038434 597 catalog_manager.cc:6719] ReplaceTablet: tablet 4200db822deb4ace936f561c230ac1fb deleted and replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40
I20251029 08:52:18.038720 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11375103cdf341e8a98dca7cca956197" } modify_external_catalogs: true
I20251029 08:52:18.038894 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11375103cdf341e8a98dca7cca956197" } modify_external_catalogs: true
I20251029 08:52:17.928488 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3ba512bd4f04bd9aceb4cb210fe9689" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:17.942160 1946 raft_consensus.cc:493] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:17.954527 1177 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.042030 1946 raft_consensus.cc:515] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.042198 1177 ts_tablet_manager.cc:1403] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.000s
I20251029 08:52:18.042631 1946 leader_election.cc:290] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.959875 2046 raft_consensus.cc:493] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.042790 2046 raft_consensus.cc:515] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.043135 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 215cd666c71a4179a19e298868a88e96 (table default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]) successfully deleted
I20251029 08:52:18.043267 2046 leader_election.cc:290] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:17.960515 861 leader_election.cc:304] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:18.043471 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 82971.8us
I20251029 08:52:18.043522 1974 tablet_replica.cc:333] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.043637 861 leader_election.cc:304] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:18.043815 1946 raft_consensus.cc:2804] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:18.043915 1946 raft_consensus.cc:697] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:18.044167 1946 consensus_queue.cc:237] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.044275 861 leader_election.cc:304] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.029559 2016 raft_consensus.cc:493] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.044615 2016 raft_consensus.cc:515] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.045048 2016 leader_election.cc:290] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:18.045087 2032 raft_consensus.cc:2804] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:18.030956 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072cf27b0743488c8166e03b5e61c545" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.045217 2032 raft_consensus.cc:697] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:18.045418 2032 consensus_queue.cc:237] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.046866 2016 raft_consensus.cc:2749] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.047412 1177 raft_consensus.cc:359] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.047564 1177 raft_consensus.cc:385] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.047621 1177 raft_consensus.cc:740] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.047803 1177 consensus_queue.cc:260] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.048247 1177 ts_tablet_manager.cc:1434] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:18.048494 1177 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.049836 1177 tablet_bootstrap.cc:654] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.038743 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4200db822deb4ace936f561c230ac1fb with delete_type TABLET_DATA_DELETED (replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:17.958191 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4e3a2fab467348099a4a1fec74eb5e65
I20251029 08:52:18.052505 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f760df7015424091acf5d9b6b39bfbd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.053925 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f760df7015424091acf5d9b6b39bfbd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:18.055177 861 leader_election.cc:343] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5a7a673ba4fd4abcbfc3493c38dcc257
I20251029 08:52:17.966002 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f0f2d3061ba74387a865d0f5f47598f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:18.055824 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:18.056265 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0f2d3061ba74387a865d0f5f47598f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:18.057889 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbf9e36c47be4509a998be8e7138475b consensus state has no leader
I20251029 08:52:18.057994 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4f80728a7a04583bada485707beb806" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:17.967561 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_684cbdb8f5aa41cca591cf8b4e991c11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:18.058485 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "608b7095d42941ccb0c4eb57f6754e74" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.058770 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_684cbdb8f5aa41cca591cf8b4e991c11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.063757 1179 ts_tablet_manager.cc:1929] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.063925 1179 log.cc:1199] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ef0a7b31ced24388bc9d9964a42fa203
I20251029 08:52:18.064291 1179 ts_tablet_manager.cc:1950] T ef0a7b31ced24388bc9d9964a42fa203 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:18.064639 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cf7604670833495c9dd61258a3301bb4
I20251029 08:52:18.065770 888 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7604670833495c9dd61258a3301bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.066605 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c706343d44a4807b1ea60c1b1e1ea49" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.066668 861 leader_election.cc:343] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:18.066826 861 leader_election.cc:343] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:18.068596 861 leader_election.cc:343] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.043712 1974 raft_consensus.cc:2243] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.072806 1974 raft_consensus.cc:2272] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.073503 1974 ts_tablet_manager.cc:1916] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:18.007145 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c32679cbb1884f09addb0f7c26ac941e consensus state has no leader
I20251029 08:52:18.075273 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a603e0ef22e48f18263ece15b799c0c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:18.084148 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_022219d62443456187fdfdf9562de3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:18.084609 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet f33c0bbf104046ccb86ff313f08da22a with delete_type TABLET_DATA_DELETED (replaced by tablet aa4f97075b3941aaa7cda3f9413e867f) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.084619 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet bc96ef1a94844a32923087c44a5bcfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:18.084825 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_022219d62443456187fdfdf9562de3f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.085350 1179 tablet_replica.cc:333] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:18.085570 1179 raft_consensus.cc:2243] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.085773 1179 raft_consensus.cc:2272] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.086447 1179 ts_tablet_manager.cc:1916] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.086681 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_247e1528f531419d80295027ead897a0" } new_table_name: "default.Table_8446ed3f2edc40a69cd466f92b5646d6" modify_external_catalogs: true
I20251029 08:52:18.088047 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_856c731e03944711af6efcd38024791e" } new_table_name: "default.Table_cbf544afceae424696ddc50a2292d48b" modify_external_catalogs: true
I20251029 08:52:18.088263 1177 tablet_bootstrap.cc:492] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.088429 1177 ts_tablet_manager.cc:1403] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20251029 08:52:18.089164 1177 raft_consensus.cc:359] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.089332 1177 raft_consensus.cc:385] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.089397 1177 raft_consensus.cc:740] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.089545 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cf4be2bf29e44a4bbb908e06223cc48d" } modify_external_catalogs: true
I20251029 08:52:18.089592 1177 consensus_queue.cc:260] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.089789 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cf4be2bf29e44a4bbb908e06223cc48d" } modify_external_catalogs: true
I20251029 08:52:18.098317 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_36384ce4abb64dd886df8435345c4a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.099093 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36384ce4abb64dd886df8435345c4a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.099364 1177 ts_tablet_manager.cc:1434] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:18.100311 1177 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.102425 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet abc307b14aff4c5e937a5151aab1dfb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.103941 1177 tablet_bootstrap.cc:654] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.105237 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 215cd666c71a4179a19e298868a88e96 with delete_type TABLET_DATA_DELETED (replaced by tablet af913080bfe44a86a76704ffa32cd9e1) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.105612 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet ee8117e04f5f40e98ea755dc461067dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.105918 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a7a673ba4fd4abcbfc3493c38dcc257 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b5051ff776e4975b917db0393a1f945) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.112118 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c5008d9df9419bb4b6d678c6dc27a6 with delete_type TABLET_DATA_DELETED (replaced by tablet afa2bb1075c4418eb10f4493f5e73c1b) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:18.113127 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0d00a6b772d4034ac7fa764884c7081 consensus state has no leader
I20251029 08:52:18.113845 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 4200db822deb4ace936f561c230ac1fb with delete_type TABLET_DATA_DELETED (replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:18.114192 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 647c637b00044033a1f6d461a67b6500 has no consensus state
I20251029 08:52:18.114228 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.117209 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7604670833495c9dd61258a3301bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.117869 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ef0a7b31ced24388bc9d9964a42fa203 (table default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]) successfully deleted
W20251029 08:52:18.117926 996 leader_election.cc:343] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:18.118041 996 leader_election.cc:304] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.118664 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_50bc8d05fa8346ebaaa0c1a6a198d929"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.119362 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50bc8d05fa8346ebaaa0c1a6a198d929 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.121562 2044 raft_consensus.cc:2749] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.121855 2044 raft_consensus.cc:493] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.122040 2044 raft_consensus.cc:515] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.117638 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3ba512bd4f04bd9aceb4cb210fe9689" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.129762 2044 leader_election.cc:290] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.017566 747 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:18.033712 720 leader_election.cc:343] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.077411 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 4200db822deb4ace936f561c230ac1fb with delete_type TABLET_DATA_DELETED (replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.077783 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.078071 745 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7604670833495c9dd61258a3301bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.134096 720 leader_election.cc:304] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.134406 2042 raft_consensus.cc:2749] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:18.134756 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 101059us
W20251029 08:52:18.135901 861 leader_election.cc:343] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e4f80728a7a04583bada485707beb806
I20251029 08:52:18.136044 861 leader_election.cc:304] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:18.136780 861 leader_election.cc:343] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:18.136875 861 leader_election.cc:304] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.137061 1946 raft_consensus.cc:2749] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.137228 1946 raft_consensus.cc:2749] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:18.137512 861 leader_election.cc:343] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2c706343d44a4807b1ea60c1b1e1ea49
I20251029 08:52:18.137620 861 leader_election.cc:304] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.137674 1177 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.137805 1946 raft_consensus.cc:2749] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.138079 1177 ts_tablet_manager.cc:1403] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
W20251029 08:52:18.078095 718 leader_election.cc:343] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.124450 1979 raft_consensus.cc:493] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.138993 1177 raft_consensus.cc:359] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.139043 1979 raft_consensus.cc:515] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.139147 1177 raft_consensus.cc:385] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.139200 1177 raft_consensus.cc:740] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.139400 1177 consensus_queue.cc:260] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.139595 1979 leader_election.cc:290] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:18.139858 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 61772.4us
I20251029 08:52:18.139897 1177 ts_tablet_manager.cc:1434] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:18.139986 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_67e2e2e244b947f9b3b19e4c352c98e9" } new_table_name: "default.Table_4d8e69bc93794c5fbfd2820e9acfedd9" modify_external_catalogs: true
I20251029 08:52:18.141407 1177 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.141968 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eed66c3fa3f848df905a2a31fd553474" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.141984 924 raft_consensus.cc:1275] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Refusing update from remote peer 7a8d80098fcb40ada903c3b2bacc2ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:18.143020 1979 raft_consensus.cc:493] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.143193 1177 tablet_bootstrap.cc:654] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.143141 1979 raft_consensus.cc:515] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:18.143714 1979 leader_election.cc:290] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:18.145839 998 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bc96ef1a94844a32923087c44a5bcfc8
W20251029 08:52:18.146121 718 leader_election.cc:343] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.146095 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc04616301354cfe873c825a9f5d7ab6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.146615 718 leader_election.cc:343] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.146544 1979 consensus_queue.cc:1048] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:18.150000 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_836882de6c804a5c908824f31f0aa328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.034171 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4cee30efd10f47c88ab3c9c5a2df5348" } modify_external_catalogs: true
I20251029 08:52:18.151798 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4cee30efd10f47c88ab3c9c5a2df5348" } modify_external_catalogs: true
I20251029 08:52:18.158567 1176 log.cc:1199] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:18.158672 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6e7b5c67ba3644c4be9ef7129ea50504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:18.159288 1176 ts_tablet_manager.cc:1950] T 363fcf8a3e0f4770a9743fddffa6495f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:18.159399 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e7b5c67ba3644c4be9ef7129ea50504 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.161010 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 691) took 1551 ms. Trace:
I20251029 08:52:18.168161 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0921613fbbaa49ae95be6d937ab40033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.168884 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0921613fbbaa49ae95be6d937ab40033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.169687 1177 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.169826 1177 ts_tablet_manager.cc:1403] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20251029 08:52:18.170579 1177 raft_consensus.cc:359] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.170769 1177 raft_consensus.cc:385] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.170835 1177 raft_consensus.cc:740] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.171067 1177 consensus_queue.cc:260] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.172425 1177 ts_tablet_manager.cc:1434] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251029 08:52:18.173326 1177 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.175122 1177 tablet_bootstrap.cc:654] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.179473 1183 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:18.179690 1183 ts_tablet_manager.cc:1403] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.269s user 0.006s sys 0.004s
I20251029 08:52:18.167550 1979 raft_consensus.cc:493] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.181995 1979 raft_consensus.cc:515] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:18.182610 1979 leader_election.cc:290] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:18.183375 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbc4345cabbc476485b6a040a076ddaf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.183936 718 leader_election.cc:343] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.187834 1176 rpcz_store.cc:276] 1029 08:52:16.609083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.609130 (+ 47us) service_pool.cc:225] Handling call
1029 08:52:17.904654 (+1295524us) ts_tablet_manager.cc:1151] Deleting tablet 363fcf8a3e0f4770a9743fddffa6495f
1029 08:52:17.904666 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.011837 (+107171us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.015100 (+ 3263us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.160983 (+145883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290914}
I20251029 08:52:18.188481 1176 tablet_replica.cc:333] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:18.188742 1176 raft_consensus.cc:2243] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.188949 1176 raft_consensus.cc:2272] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.189507 1176 ts_tablet_manager.cc:1916] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.180794 1178 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:18.193116 1178 ts_tablet_manager.cc:1403] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.205s user 0.003s sys 0.007s
I20251029 08:52:18.194248 1178 raft_consensus.cc:359] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.194501 1178 raft_consensus.cc:385] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.194629 1178 raft_consensus.cc:740] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.195027 1178 consensus_queue.cc:260] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.195694 1178 ts_tablet_manager.cc:1434] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:18.195993 1178 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:18.201465 1183 raft_consensus.cc:359] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.201757 1183 raft_consensus.cc:385] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.201870 1183 raft_consensus.cc:740] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.202132 1183 consensus_queue.cc:260] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.202674 1183 ts_tablet_manager.cc:1434] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251029 08:52:18.203225 1183 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:18.207005 2098 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:18.212051 2099 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:18.213711 2104 tablet.cc:1722] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:18.220331 2098 tablet.cc:1722] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 08:52:18.222208 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b1af4bcd7af49a0b8a419298ab37ea9 consensus state has no leader
I20251029 08:52:18.225023 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:18.225617 1974 ts_tablet_manager.cc:1929] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.227463 745 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.235105 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0195d09f103745aca4e75dfbeb05f244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.235881 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0195d09f103745aca4e75dfbeb05f244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.237462 1179 ts_tablet_manager.cc:1929] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.238744 1179 log.cc:1199] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/14b665b63a7341d699d499ffc8cfd9f7
I20251029 08:52:18.248932 1178 tablet_bootstrap.cc:654] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.255538 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 65eaf12e45c54bd1b0aedc647dc90436 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:18.256671 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a603e0ef22e48f18263ece15b799c0c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:18.259354 1183 tablet_bootstrap.cc:654] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.244521 1179 ts_tablet_manager.cc:1950] T 14b665b63a7341d699d499ffc8cfd9f7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:18.261993 1179 tablet_replica.cc:333] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:18.262341 1179 raft_consensus.cc:2243] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.262642 1179 raft_consensus.cc:2272] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.263499 1179 ts_tablet_manager.cc:1916] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.268580 2104 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:18.226266 1177 tablet_bootstrap.cc:492] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.270125 1974 log.cc:1199] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/757e34137e884d5ca5cd88fdd03244d8
I20251029 08:52:18.242179 2098 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:18.270177 1177 ts_tablet_manager.cc:1403] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.005s
I20251029 08:52:18.275683 2104 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:18.285389 1974 ts_tablet_manager.cc:1950] T 757e34137e884d5ca5cd88fdd03244d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.286463 1177 raft_consensus.cc:359] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.286640 1177 raft_consensus.cc:385] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.286695 1177 raft_consensus.cc:740] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.287043 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1f00f2933c364d019c2b2d749778ad6d" } new_table_name: "default.Table_02acde1d2bfe41e9a0f7565ed487c1c7" modify_external_catalogs: true
I20251029 08:52:18.288311 1178 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:18.289465 1178 ts_tablet_manager.cc:1403] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.002s
W20251029 08:52:18.296924 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85a806cc04e641d9a1c4af97a24b13de consensus state has no leader
I20251029 08:52:18.297797 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f40b8aaae0b34603be167b08d72d08d3" } modify_external_catalogs: true
I20251029 08:52:18.298067 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f40b8aaae0b34603be167b08d72d08d3" } modify_external_catalogs: true
I20251029 08:52:18.298519 2099 tablet.cc:1722] T 63dfcd944c1f487b8911ae383770d930 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:18.286888 1177 consensus_queue.cc:260] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.301316 1177 ts_tablet_manager.cc:1434] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251029 08:52:18.303026 1177 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.303802 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 704) took 1578 ms. Trace:
I20251029 08:52:18.303920 1974 rpcz_store.cc:276] 1029 08:52:16.725632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.726143 (+ 511us) service_pool.cc:225] Handling call
1029 08:52:18.043466 (+1317323us) ts_tablet_manager.cc:1151] Deleting tablet 757e34137e884d5ca5cd88fdd03244d8
1029 08:52:18.043479 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.167318 (+123839us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.225517 (+ 58199us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.303781 (+ 78264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316614}
I20251029 08:52:18.303994 1178 raft_consensus.cc:359] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.304495 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 757e34137e884d5ca5cd88fdd03244d8 (table default.Table_4a022119f31d4460af5cdd68baa8521b [id=9eab4fb7fff44c10854a84cbf994f667]) successfully deleted
I20251029 08:52:18.304898 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 706) took 1518 ms. Trace:
I20251029 08:52:18.305017 1974 rpcz_store.cc:276] 1029 08:52:16.786764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.786832 (+ 68us) service_pool.cc:225] Handling call
1029 08:52:18.304776 (+1517944us) ts_tablet_manager.cc:1151] Deleting tablet c1f6ca7d4e534a46a1fba75acd7df39e
1029 08:52:18.304786 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.304881 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517734}
W20251029 08:52:18.313233 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c1f6ca7d4e534a46a1fba75acd7df39e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f6ca7d4e534a46a1fba75acd7df39e already in progress: creating tablet
I20251029 08:52:18.312772 1177 tablet_bootstrap.cc:654] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.314275 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 709) took 1312 ms. Trace:
I20251029 08:52:18.314433 1974 rpcz_store.cc:276] 1029 08:52:17.001587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.004069 (+ 2482us) service_pool.cc:225] Handling call
1029 08:52:18.314162 (+1310093us) ts_tablet_manager.cc:1151] Deleting tablet 116cc0d155e24653937953eea4f39ee4
1029 08:52:18.314173 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.314257 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309931}
W20251029 08:52:18.314920 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 116cc0d155e24653937953eea4f39ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116cc0d155e24653937953eea4f39ee4 already in progress: creating tablet
I20251029 08:52:18.315285 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 727) took 1050 ms. Trace:
I20251029 08:52:18.315415 1974 rpcz_store.cc:276] 1029 08:52:17.265211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.265274 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:18.315182 (+1049908us) ts_tablet_manager.cc:1151] Deleting tablet 29bb35cd36464dbd96470cc78faa481f
1029 08:52:18.315192 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.315271 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049671}
W20251029 08:52:18.315821 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 29bb35cd36464dbd96470cc78faa481f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29bb35cd36464dbd96470cc78faa481f already in progress: creating tablet
I20251029 08:52:18.316740 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 738) took 1049 ms. Trace:
I20251029 08:52:18.316851 1974 rpcz_store.cc:276] 1029 08:52:17.267111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.628898 (+361787us) service_pool.cc:225] Handling call
1029 08:52:18.316650 (+687752us) ts_tablet_manager.cc:1151] Deleting tablet fd4ac3f024a74eb592a1de6fd8f57985
1029 08:52:18.316657 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.316725 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":687493}
W20251029 08:52:18.317323 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fd4ac3f024a74eb592a1de6fd8f57985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd4ac3f024a74eb592a1de6fd8f57985 already in progress: creating tablet
I20251029 08:52:18.317701 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 739) took 1050 ms. Trace:
I20251029 08:52:18.317813 1974 rpcz_store.cc:276] 1029 08:52:17.267195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.629192 (+361997us) service_pool.cc:225] Handling call
1029 08:52:18.317605 (+688413us) ts_tablet_manager.cc:1151] Deleting tablet 98324d4ae1af45fa94ae438dc2a64aaa
1029 08:52:18.317613 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.317683 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":688269}
W20251029 08:52:18.318221 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 98324d4ae1af45fa94ae438dc2a64aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98324d4ae1af45fa94ae438dc2a64aaa already in progress: creating tablet
I20251029 08:52:18.318558 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 741) took 1051 ms. Trace:
I20251029 08:52:18.318665 1974 rpcz_store.cc:276] 1029 08:52:17.267364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.631060 (+363696us) service_pool.cc:225] Handling call
1029 08:52:18.318470 (+687410us) ts_tablet_manager.cc:1151] Deleting tablet 6689cf0804f946c18ebc555a191549d4
1029 08:52:18.318479 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.318545 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":687227}
W20251029 08:52:18.319134 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6689cf0804f946c18ebc555a191549d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6689cf0804f946c18ebc555a191549d4 already in progress: creating tablet
I20251029 08:52:18.320092 1178 raft_consensus.cc:385] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.320240 1178 raft_consensus.cc:740] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.314002 1176 ts_tablet_manager.cc:1929] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.320585 1176 log.cc:1199] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/fd25eeb89e574297a92d8e978a79d768
I20251029 08:52:18.321269 1176 ts_tablet_manager.cc:1950] T fd25eeb89e574297a92d8e978a79d768 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:18.322144 1183 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:18.322197 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 692) took 1713 ms. Trace:
I20251029 08:52:18.322281 1176 rpcz_store.cc:276] 1029 08:52:16.609208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.609291 (+ 83us) service_pool.cc:225] Handling call
1029 08:52:18.188434 (+1579143us) ts_tablet_manager.cc:1151] Deleting tablet fd25eeb89e574297a92d8e978a79d768
1029 08:52:18.188444 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.275495 (+ 87051us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.313898 (+ 38403us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.322175 (+ 8277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574633}
I20251029 08:52:18.322393 1183 ts_tablet_manager.cc:1403] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.000s
I20251029 08:52:18.320819 1178 consensus_queue.cc:260] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.323333 1178 ts_tablet_manager.cc:1434] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20251029 08:52:18.323611 1178 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:18.325345 1178 tablet_bootstrap.cc:654] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.329157 1183 raft_consensus.cc:359] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.329690 1183 raft_consensus.cc:385] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.335955 1183 raft_consensus.cc:740] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.325603 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 693) took 1666 ms. Trace:
I20251029 08:52:18.336410 1176 rpcz_store.cc:276] 1029 08:52:16.659298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.659373 (+ 75us) service_pool.cc:225] Handling call
1029 08:52:18.325471 (+1666098us) ts_tablet_manager.cc:1151] Deleting tablet fc78a202be244c3e9ad354bead9bfe52
1029 08:52:18.325481 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.325586 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665919}
I20251029 08:52:18.336439 1183 consensus_queue.cc:260] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.337325 1176 tablet_replica.cc:333] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:18.337594 1176 raft_consensus.cc:2243] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.322664 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet fd25eeb89e574297a92d8e978a79d768 (table default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]) successfully deleted
I20251029 08:52:18.337966 1176 raft_consensus.cc:2272] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:18.338085 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fc78a202be244c3e9ad354bead9bfe52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc78a202be244c3e9ad354bead9bfe52 already in progress: creating tablet
I20251029 08:52:18.338449 1183 ts_tablet_manager.cc:1434] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:18.338843 1176 ts_tablet_manager.cc:1916] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.340622 1179 ts_tablet_manager.cc:1929] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.348114 1179 log.cc:1199] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ba1bcb3f6c134de2b322696501927a16
I20251029 08:52:18.348546 1179 ts_tablet_manager.cc:1950] T ba1bcb3f6c134de2b322696501927a16 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:18.349529 1179 tablet_replica.cc:333] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:18.350200 1974 tablet_replica.cc:333] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.350431 1974 raft_consensus.cc:2243] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:18.350908 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d306be1d51674c2abc506d248e05a9c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d306be1d51674c2abc506d248e05a9c7 already in progress: creating tablet
I20251029 08:52:18.352030 1179 raft_consensus.cc:2243] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.352232 1179 raft_consensus.cc:2272] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.352427 1974 raft_consensus.cc:2272] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.353034 1974 ts_tablet_manager.cc:1916] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.354835 2042 raft_consensus.cc:493] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.354961 2042 raft_consensus.cc:515] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.355535 2042 leader_election.cc:290] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.356262 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c66cbb495f5142d79a674f7d5adc505e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.349648 1183 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:18.357976 1183 tablet_bootstrap.cc:654] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.359568 1178 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:18.359757 1178 ts_tablet_manager.cc:1403] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
W20251029 08:52:18.360958 718 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.361351 1178 raft_consensus.cc:359] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.361616 1178 raft_consensus.cc:385] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.361722 1178 raft_consensus.cc:740] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.362017 1178 consensus_queue.cc:260] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.362746 1178 ts_tablet_manager.cc:1434] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:18.363595 1178 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:18.365895 1178 tablet_bootstrap.cc:654] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.366417 1179 ts_tablet_manager.cc:1916] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:18.373167 996 leader_election.cc:343] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.377084 996 leader_election.cc:304] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.377696 2050 raft_consensus.cc:2749] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:18.378278 720 leader_election.cc:343] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.378407 720 leader_election.cc:304] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.378746 2042 raft_consensus.cc:2749] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.386442 1974 ts_tablet_manager.cc:1929] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.386910 1176 ts_tablet_manager.cc:1929] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.390731 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 14b665b63a7341d699d499ffc8cfd9f7 (table default.Table_63c241868b434e1d9af341e235b0554f [id=d634f975379740feb85479b2e5c8607e]) successfully deleted
I20251029 08:52:18.391475 1176 log.cc:1199] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:18.391825 1176 ts_tablet_manager.cc:1950] T d30183814f5e4f2ab9d71fe493eb801c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:18.392040 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eed66c3fa3f848df905a2a31fd553474" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.392527 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc04616301354cfe873c825a9f5d7ab6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.392807 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ba1bcb3f6c134de2b322696501927a16 (table default.Table_76059d1e191546928659040fec3bdf30 [id=beac13de5386419c8239948d6af5ed23]) successfully deleted
I20251029 08:52:18.393654 2050 consensus_queue.cc:1048] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:18.396482 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbc4345cabbc476485b6a040a076ddaf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.398006 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 694) took 1707 ms. Trace:
I20251029 08:52:18.398231 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c66cbb495f5142d79a674f7d5adc505e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.399133 996 leader_election.cc:343] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:18.400151 996 leader_election.cc:304] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.400637 2050 raft_consensus.cc:2749] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.402987 592 catalog_manager.cc:5649] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:18.403447 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:18.403694 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:18.403905 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:18.407409 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:18.412674 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8446ed3f2edc40a69cd466f92b5646d6" } new_table_name: "default.Table_4ff1bcc056504a13864a32f4f16bceb9" modify_external_catalogs: true
I20251029 08:52:18.415120 588 catalog_manager.cc:5649] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:18.415591 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:18.408684 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 084a659932ec4996a6d6b160d67efdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.419108 1974 log.cc:1199] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ef0a7b31ced24388bc9d9964a42fa203
I20251029 08:52:18.419564 1974 ts_tablet_manager.cc:1950] T ef0a7b31ced24388bc9d9964a42fa203 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.420401 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ef0a7b31ced24388bc9d9964a42fa203 (table default.Table_ec3c62a3fcad44a1ae7a74fbaefc4192 [id=056d29b6cb8a48c6acac38c6868c5c4d]) successfully deleted
I20251029 08:52:18.420802 1974 tablet_replica.cc:333] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.421046 1974 raft_consensus.cc:2243] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.421283 1974 raft_consensus.cc:2272] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.421959 1974 ts_tablet_manager.cc:1916] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.421944 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:18.424595 1183 tablet_bootstrap.cc:492] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:18.424794 1183 ts_tablet_manager.cc:1403] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20251029 08:52:18.425621 1183 raft_consensus.cc:359] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.425843 1183 raft_consensus.cc:385] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.425926 1183 raft_consensus.cc:740] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.426189 1183 consensus_queue.cc:260] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.428273 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 391977048e5f4f0b8f62e4d5e3d2946b
I20251029 08:52:18.429626 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61e96f14b6c74450828ef50148ab7ac5
I20251029 08:52:18.429627 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.430534 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2f36641b7086430a89380056e04b8aaf
I20251029 08:52:18.433118 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 65eaf12e45c54bd1b0aedc647dc90436
I20251029 08:52:18.433615 593 catalog_manager.cc:6719] ReplaceTablet: tablet 65eaf12e45c54bd1b0aedc647dc90436 deleted and replaced by tablet 295ab2a10a5648abb59bc045c7dbe165
I20251029 08:52:18.437201 1183 ts_tablet_manager.cc:1434] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20251029 08:52:18.439136 1183 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:18.441720 1183 tablet_bootstrap.cc:654] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.441917 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f36641b7086430a89380056e04b8aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.442135 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 with delete_type TABLET_DATA_DELETED (replaced by tablet 295ab2a10a5648abb59bc045c7dbe165) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.442281 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:18.398682 720 consensus_peers.cc:597] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:18.445824 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 47168.4us
W20251029 08:52:18.446075 720 leader_election.cc:343] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.446182 720 leader_election.cc:304] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:18.446596 720 leader_election.cc:343] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.446682 720 leader_election.cc:304] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:18.446971 720 leader_election.cc:343] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.447048 720 leader_election.cc:304] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.447542 2038 raft_consensus.cc:2749] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.447697 2038 raft_consensus.cc:2749] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.447804 2038 raft_consensus.cc:2749] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.408862 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 084a659932ec4996a6d6b160d67efdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.432665 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.432838 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.435842 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f36641b7086430a89380056e04b8aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.440155 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 with delete_type TABLET_DATA_DELETED (replaced by tablet 295ab2a10a5648abb59bc045c7dbe165) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.448880 1177 tablet_bootstrap.cc:492] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.449083 1177 ts_tablet_manager.cc:1403] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.146s user 0.006s sys 0.010s
I20251029 08:52:18.449923 1177 raft_consensus.cc:359] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.450235 1177 raft_consensus.cc:385] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.450369 1177 raft_consensus.cc:740] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.450717 1177 consensus_queue.cc:260] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.451874 1177 ts_tablet_manager.cc:1434] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:18.453437 1177 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.460873 652 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61e96f14b6c74450828ef50148ab7ac5
I20251029 08:52:18.462455 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.463804 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.466794 888 tablet_service.cc:1505] Processing CreateTablet for tablet 211f3fde17f64c99a0b80e377c028fb4 (DEFAULT_TABLE table=default.Table_6bfec7dc813f42d588da6bd24fe2f3b7 [id=2d2782928dab44969c910f4769f1fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.467140 1177 tablet_bootstrap.cc:654] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.467363 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f3fde17f64c99a0b80e377c028fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.469535 1176 rpcz_store.cc:276] 1029 08:52:16.690336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.690391 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:18.337273 (+1646882us) ts_tablet_manager.cc:1151] Deleting tablet d30183814f5e4f2ab9d71fe493eb801c
1029 08:52:18.337283 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.346180 (+ 8897us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.386819 (+ 40639us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.397988 (+ 11169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646696}
I20251029 08:52:18.470103 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d30183814f5e4f2ab9d71fe493eb801c (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
W20251029 08:52:18.471427 996 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 70812.5us
I20251029 08:52:18.474684 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 695) took 1780 ms. Trace:
I20251029 08:52:18.474803 1176 rpcz_store.cc:276] 1029 08:52:16.694613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.695249 (+ 636us) service_pool.cc:225] Handling call
1029 08:52:18.474477 (+1779228us) ts_tablet_manager.cc:1151] Deleting tablet 64058f9cb3ad4844ac87bc0cc27774b9
1029 08:52:18.474494 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.474661 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778987}
W20251029 08:52:18.475366 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 64058f9cb3ad4844ac87bc0cc27774b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64058f9cb3ad4844ac87bc0cc27774b9 already in progress: creating tablet
W20251029 08:52:18.479686 720 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.480525 748 tablet_service.cc:1505] Processing CreateTablet for tablet 211f3fde17f64c99a0b80e377c028fb4 (DEFAULT_TABLE table=default.Table_6bfec7dc813f42d588da6bd24fe2f3b7 [id=2d2782928dab44969c910f4769f1fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.480619 1974 ts_tablet_manager.cc:1929] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.480774 1974 log.cc:1199] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:18.481009 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f3fde17f64c99a0b80e377c028fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.481141 1974 ts_tablet_manager.cc:1950] T a20f26d0f7874ccb960ffe4d27e854e1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.482080 1974 tablet_replica.cc:333] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.482259 1974 raft_consensus.cc:2243] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.482283 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a20f26d0f7874ccb960ffe4d27e854e1 (table default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]) successfully deleted
I20251029 08:52:18.482402 1974 raft_consensus.cc:2272] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.482898 1974 ts_tablet_manager.cc:1916] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.483695 896 tablet_service.cc:1505] Processing CreateTablet for tablet aa30a26ea96a4621b5382d0db40427b3 (DEFAULT_TABLE table=default.Table_124a72a977e04bf391ade81da1a92c48 [id=8e102fe1ca0a498a9efdc8b025278f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.484133 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa30a26ea96a4621b5382d0db40427b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.488477 720 leader_election.cc:304] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.488750 2038 raft_consensus.cc:2749] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.489239 2050 raft_consensus.cc:493] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.489387 2050 raft_consensus.cc:515] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.489758 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 084a659932ec4996a6d6b160d67efdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.489987 2050 leader_election.cc:290] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.490095 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.490379 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.490694 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f36641b7086430a89380056e04b8aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:18.490973 998 leader_election.cc:343] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c7024e6eb4124109ad1575d216739fe2
I20251029 08:52:18.490988 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 with delete_type TABLET_DATA_DELETED (replaced by tablet 295ab2a10a5648abb59bc045c7dbe165) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.491170 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.491391 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.491772 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 211f3fde17f64c99a0b80e377c028fb4 (DEFAULT_TABLE table=default.Table_6bfec7dc813f42d588da6bd24fe2f3b7 [id=2d2782928dab44969c910f4769f1fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.491937 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7024e6eb4124109ad1575d216739fe2" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.492172 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f3fde17f64c99a0b80e377c028fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.496167 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.496356 1178 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:18.496511 1178 ts_tablet_manager.cc:1403] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.133s user 0.005s sys 0.000s
I20251029 08:52:18.497341 1178 raft_consensus.cc:359] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.497958 1178 raft_consensus.cc:385] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.498093 1178 raft_consensus.cc:740] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.498378 1178 consensus_queue.cc:260] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.499315 1178 ts_tablet_manager.cc:1434] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:18.499655 1178 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:18.502717 1177 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.502717 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 696) took 1808 ms. Trace:
I20251029 08:52:18.502867 1177 ts_tablet_manager.cc:1403] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.002s
I20251029 08:52:18.502791 1176 rpcz_store.cc:276] 1029 08:52:16.694730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.695519 (+ 789us) service_pool.cc:225] Handling call
1029 08:52:18.502586 (+1807067us) ts_tablet_manager.cc:1151] Deleting tablet 24834985a71349b9865840e9b5537d72
1029 08:52:18.502596 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.502698 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806925}
I20251029 08:52:18.503093 1176 tablet_replica.cc:333] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:18.503260 1176 raft_consensus.cc:2243] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:18.503283 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 24834985a71349b9865840e9b5537d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24834985a71349b9865840e9b5537d72 already in progress: creating tablet
I20251029 08:52:18.503407 1176 raft_consensus.cc:2272] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.503626 1177 raft_consensus.cc:359] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:18.503785 1177 raft_consensus.cc:385] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.503831 1177 raft_consensus.cc:740] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.503909 1176 ts_tablet_manager.cc:1916] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.504264 1177 consensus_queue.cc:260] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:18.507125 1177 ts_tablet_manager.cc:1434] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:18.507797 1177 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.509624 1177 tablet_bootstrap.cc:654] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.512704 748 tablet_service.cc:1505] Processing CreateTablet for tablet aa30a26ea96a4621b5382d0db40427b3 (DEFAULT_TABLE table=default.Table_124a72a977e04bf391ade81da1a92c48 [id=8e102fe1ca0a498a9efdc8b025278f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.513166 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa30a26ea96a4621b5382d0db40427b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.513170 1946 raft_consensus.cc:493] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.513327 1946 raft_consensus.cc:515] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.513893 1946 leader_election.cc:290] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:18.514724 861 leader_election.cc:343] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 072cf27b0743488c8166e03b5e61c545
I20251029 08:52:18.514737 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072cf27b0743488c8166e03b5e61c545" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.514910 1070 raft_consensus.cc:2468] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:18.515319 861 leader_election.cc:304] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.515552 1946 raft_consensus.cc:2804] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:18.515633 1946 raft_consensus.cc:493] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:18.515702 1946 raft_consensus.cc:3060] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:18.517612 996 leader_election.cc:343] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.517866 996 leader_election.cc:304] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.518122 2050 raft_consensus.cc:2749] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.519368 1946 raft_consensus.cc:515] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.519974 1946 leader_election.cc:290] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:18.520314 1177 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.520381 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072cf27b0743488c8166e03b5e61c545" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:18.520488 1177 ts_tablet_manager.cc:1403] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20251029 08:52:18.520565 1070 raft_consensus.cc:3060] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.521279 1179 ts_tablet_manager.cc:1929] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.521217 1177 raft_consensus.cc:359] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.521364 1179 log.cc:1199] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3295bb0b47714080a5c3f34049acd487
I20251029 08:52:18.521366 1177 raft_consensus.cc:385] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.521430 1177 raft_consensus.cc:740] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.521653 1179 ts_tablet_manager.cc:1950] T 3295bb0b47714080a5c3f34049acd487 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:18.521652 1177 consensus_queue.cc:260] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.522104 1177 ts_tablet_manager.cc:1434] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:18.522882 1177 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.524246 1045 tablet_service.cc:1505] Processing CreateTablet for tablet aa30a26ea96a4621b5382d0db40427b3 (DEFAULT_TABLE table=default.Table_124a72a977e04bf391ade81da1a92c48 [id=8e102fe1ca0a498a9efdc8b025278f89]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:18.524405 861 leader_election.cc:343] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 072cf27b0743488c8166e03b5e61c545
I20251029 08:52:18.524742 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa30a26ea96a4621b5382d0db40427b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.524955 1177 tablet_bootstrap.cc:654] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.527432 1036 tablet_service.cc:1505] Processing CreateTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 (DEFAULT_TABLE table=default.Table_343f0a80fc4c4dceb7098e5f75a81389 [id=b0d6b62befe0433da248cc1d91956228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.527760 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a91ac12e46ca9bc15e3bb9d39a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.530906 888 tablet_service.cc:1505] Processing CreateTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 (DEFAULT_TABLE table=default.Table_343f0a80fc4c4dceb7098e5f75a81389 [id=b0d6b62befe0433da248cc1d91956228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.531698 1974 ts_tablet_manager.cc:1929] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.531876 1974 log.cc:1199] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4e2773cae9814df79ee60ea8b0fe4e9d
I20251029 08:52:18.532279 1974 ts_tablet_manager.cc:1950] T 4e2773cae9814df79ee60ea8b0fe4e9d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.532706 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a91ac12e46ca9bc15e3bb9d39a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.532663 747 tablet_service.cc:1505] Processing CreateTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 (DEFAULT_TABLE table=default.Table_343f0a80fc4c4dceb7098e5f75a81389 [id=b0d6b62befe0433da248cc1d91956228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.533123 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a91ac12e46ca9bc15e3bb9d39a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.533495 2050 raft_consensus.cc:493] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.533643 2050 raft_consensus.cc:515] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.534198 2050 leader_election.cc:290] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.538786 1178 tablet_bootstrap.cc:654] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.539465 1179 tablet_replica.cc:333] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:18.533780 1974 tablet_replica.cc:333] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.540225 1974 raft_consensus.cc:2243] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.540551 1974 raft_consensus.cc:2272] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.540738 1179 raft_consensus.cc:2243] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:18.541388 1974 ts_tablet_manager.cc:1916] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.541817 1179 pending_rounds.cc:70] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:18.542006 1179 pending_rounds.cc:77] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037633810800640 op_type: NO_OP noop_request { }
W20251029 08:52:18.540825 998 leader_election.cc:343] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:18.543181 1179 raft_consensus.cc:2889] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:18.543660 1179 raft_consensus.cc:2272] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.544652 1179 ts_tablet_manager.cc:1916] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.545743 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4e2773cae9814df79ee60ea8b0fe4e9d (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:18.546592 1070 raft_consensus.cc:2468] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:18.549386 861 leader_election.cc:304] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.549733 1946 raft_consensus.cc:2804] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:18.549894 1946 raft_consensus.cc:697] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:18.550153 1946 consensus_queue.cc:237] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.556624 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c3e50018f414f4b88e0e5d90cb7c925" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.556753 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3295bb0b47714080a5c3f34049acd487 (table default.Table_32228c67a14f44429dd7a7bbed1e1e34 [id=cff295ca7fcc4342a9ec9d741f03b164]) successfully deleted
W20251029 08:52:18.557719 996 leader_election.cc:343] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.557857 996 leader_election.cc:304] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.559223 896 tablet_service.cc:1505] Processing CreateTablet for tablet afb629d515db49c69cb8a51e2f685996 (DEFAULT_TABLE table=default.Table_2f8ccb5f14eb48eb87e67d8d788ca18a [id=87cc00ea5e624a178304e3f4fc09d8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.559710 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb629d515db49c69cb8a51e2f685996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.565042 1176 ts_tablet_manager.cc:1929] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.565198 1177 tablet_bootstrap.cc:492] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:18.565382 1177 ts_tablet_manager.cc:1403] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.009s
I20251029 08:52:18.567790 2050 raft_consensus.cc:2749] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.568037 2050 raft_consensus.cc:493] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.568148 2050 raft_consensus.cc:515] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.568733 2050 leader_election.cc:290] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:18.570101 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.570305 922 raft_consensus.cc:2468] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:18.570705 996 leader_election.cc:304] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:18.570703 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:18.572314 784 raft_consensus.cc:2468] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:18.572712 2050 raft_consensus.cc:2804] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:18.572789 2050 raft_consensus.cc:493] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:18.572855 2050 raft_consensus.cc:3060] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.573000 1176 log.cc:1199] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/840ff254244f43a18679232279e02d1f
I20251029 08:52:18.573617 1176 ts_tablet_manager.cc:1950] T 840ff254244f43a18679232279e02d1f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:18.566131 1177 raft_consensus.cc:359] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.574163 1177 raft_consensus.cc:385] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.574242 1177 raft_consensus.cc:740] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.574462 1177 consensus_queue.cc:260] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:18.575039 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0d00a6b772d4034ac7fa764884c7081 consensus state has no leader
I20251029 08:52:18.575193 1177 ts_tablet_manager.cc:1434] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:18.575443 1177 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:18.575696 2050 raft_consensus.cc:515] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.577342 1177 tablet_bootstrap.cc:654] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.578336 2050 leader_election.cc:290] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:18.579254 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:18.579475 922 raft_consensus.cc:3060] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.581638 1036 tablet_service.cc:1505] Processing CreateTablet for tablet afb629d515db49c69cb8a51e2f685996 (DEFAULT_TABLE table=default.Table_2f8ccb5f14eb48eb87e67d8d788ca18a [id=87cc00ea5e624a178304e3f4fc09d8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.582012 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb629d515db49c69cb8a51e2f685996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.582074 1974 ts_tablet_manager.cc:1929] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.582188 1974 log.cc:1199] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:18.582515 922 raft_consensus.cc:2468] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:18.578697 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29adc6d53af5419dbda1d41d75d37e71" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:18.585011 784 raft_consensus.cc:3060] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.586030 996 leader_election.cc:304] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:18.582525 1974 ts_tablet_manager.cc:1950] T a31847928db44bd4b1ebd50c5130065f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.587857 784 raft_consensus.cc:2468] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:18.589057 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a31847928db44bd4b1ebd50c5130065f (table default.Table_3f1b48395900452b8b149c6aef010592 [id=fd07f1c36584477caf673f91f2a86f58]) successfully deleted
I20251029 08:52:18.593441 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:18.595793 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_09655491a5f442aaa477ef4bbb1303c1" } new_table_name: "default.Table_f130330a4317476bac6db939775b82fb" modify_external_catalogs: true
I20251029 08:52:18.597676 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6d2760f9ad334421ba99b2e63ae70e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.598408 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d2760f9ad334421ba99b2e63ae70e0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.600135 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7501d57fa16e41269af41cad333961d2" } modify_external_catalogs: true
I20251029 08:52:18.600345 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7501d57fa16e41269af41cad333961d2" } modify_external_catalogs: true
I20251029 08:52:18.600418 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fc7b6408d15e4c2a89eb2454f772395b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.600981 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc7b6408d15e4c2a89eb2454f772395b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.599853 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c2da7feabcf14a2f9d60e558d438633f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.602020 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2da7feabcf14a2f9d60e558d438633f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.600139 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6cc56e5870cf4447ab15779b3025731b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.603978 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cc56e5870cf4447ab15779b3025731b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.604336 1974 tablet_replica.cc:333] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.604655 1974 raft_consensus.cc:2243] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.604893 1974 raft_consensus.cc:2272] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.605674 1974 ts_tablet_manager.cc:1916] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.605681 757 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbb50ca480d41b095573f916bf93c30 (DEFAULT_TABLE table=default.Table_2c055a320a224f7fa1d6544e49d9989f [id=3930982762f3402997929f334494c18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.606113 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbb50ca480d41b095573f916bf93c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.608353 749 tablet_service.cc:1505] Processing CreateTablet for tablet 10162721b26941cc95ac338edec66054 (DEFAULT_TABLE table=default.Table_a7e2b6a295bc41f5885bd96998bd2723 [id=89419220fc0b47c6baf3528e771aea11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.608750 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10162721b26941cc95ac338edec66054. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.612511 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5d96adac7a4fd982d9e574b89c2982 (DEFAULT_TABLE table=default.Table_d7e9b2817baa4c87a5adad8a8905c6ef [id=f738b848eae64e4da114bbc7c2a2588c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.612859 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5d96adac7a4fd982d9e574b89c2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.625141 2050 raft_consensus.cc:2804] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:18.625372 2050 raft_consensus.cc:697] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:18.625638 2050 consensus_queue.cc:237] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:18.632737 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 46732.6us
I20251029 08:52:18.633991 1035 tablet_service.cc:1505] Processing CreateTablet for tablet e14d05e64689423da77d6d60b760cc06 (DEFAULT_TABLE table=default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71 [id=89283190678041a089bf5a813de0152d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.634399 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d05e64689423da77d6d60b760cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.634481 1974 ts_tablet_manager.cc:1929] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.634609 1974 log.cc:1199] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e4f80728a7a04583bada485707beb806
I20251029 08:52:18.634846 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 10162721b26941cc95ac338edec66054 (DEFAULT_TABLE table=default.Table_a7e2b6a295bc41f5885bd96998bd2723 [id=89419220fc0b47c6baf3528e771aea11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.634936 1974 ts_tablet_manager.cc:1950] T e4f80728a7a04583bada485707beb806 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.635198 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10162721b26941cc95ac338edec66054. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.635730 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 747) took 1015 ms. Trace:
I20251029 08:52:18.635823 1974 rpcz_store.cc:276] 1029 08:52:17.619937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.631702 (+ 11765us) service_pool.cc:225] Handling call
1029 08:52:18.604222 (+972520us) ts_tablet_manager.cc:1151] Deleting tablet e4f80728a7a04583bada485707beb806
1029 08:52:18.604242 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.617402 (+ 13160us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.634327 (+ 16925us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.635689 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":972405}
I20251029 08:52:18.640379 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5d96adac7a4fd982d9e574b89c2982 (DEFAULT_TABLE table=default.Table_d7e9b2817baa4c87a5adad8a8905c6ef [id=f738b848eae64e4da114bbc7c2a2588c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.640759 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5d96adac7a4fd982d9e574b89c2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.644722 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbb50ca480d41b095573f916bf93c30 (DEFAULT_TABLE table=default.Table_2c055a320a224f7fa1d6544e49d9989f [id=3930982762f3402997929f334494c18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.646065 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbb50ca480d41b095573f916bf93c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.648000 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 6d9face4d3c64b8987085cd5b8662db9 (DEFAULT_TABLE table=default.Table_c8619310a8224be58710ffee5aa94c98 [id=4703c4dbffee4b3098c563c6d304c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.648409 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d9face4d3c64b8987085cd5b8662db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.654405 748 tablet_service.cc:1505] Processing CreateTablet for tablet e14d05e64689423da77d6d60b760cc06 (DEFAULT_TABLE table=default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71 [id=89283190678041a089bf5a813de0152d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.654982 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d05e64689423da77d6d60b760cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.656263 1974 tablet_replica.cc:333] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.656590 1974 raft_consensus.cc:2243] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.656858 1974 raft_consensus.cc:2272] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.657528 1974 ts_tablet_manager.cc:1916] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.664393 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 850aee25ae1940bcbe6677dcea2e7b1d (DEFAULT_TABLE table=default.Table_4f68e8291cd64391bdb236f948dd338b [id=4d90d1e2a9af404a9568241d66028332]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.664433 757 tablet_service.cc:1505] Processing CreateTablet for tablet 850aee25ae1940bcbe6677dcea2e7b1d (DEFAULT_TABLE table=default.Table_4f68e8291cd64391bdb236f948dd338b [id=4d90d1e2a9af404a9568241d66028332]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.664925 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 850aee25ae1940bcbe6677dcea2e7b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.664980 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 850aee25ae1940bcbe6677dcea2e7b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.672354 747 tablet_service.cc:1505] Processing CreateTablet for tablet afb629d515db49c69cb8a51e2f685996 (DEFAULT_TABLE table=default.Table_2f8ccb5f14eb48eb87e67d8d788ca18a [id=87cc00ea5e624a178304e3f4fc09d8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.672876 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb629d515db49c69cb8a51e2f685996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.674082 758 tablet_service.cc:1505] Processing CreateTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 (DEFAULT_TABLE table=default.Table_f40b8aaae0b34603be167b08d72d08d3 [id=452d1fa58caf4ea6b9aed2783fd1caad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.674503 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e96f14b6c74450828ef50148ab7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.681841 888 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbb50ca480d41b095573f916bf93c30 (DEFAULT_TABLE table=default.Table_2c055a320a224f7fa1d6544e49d9989f [id=3930982762f3402997929f334494c18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.682310 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbb50ca480d41b095573f916bf93c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.690763 896 tablet_service.cc:1505] Processing CreateTablet for tablet 10162721b26941cc95ac338edec66054 (DEFAULT_TABLE table=default.Table_a7e2b6a295bc41f5885bd96998bd2723 [id=89419220fc0b47c6baf3528e771aea11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.691243 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10162721b26941cc95ac338edec66054. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.693672 1946 raft_consensus.cc:493] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.693809 1946 raft_consensus.cc:515] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.694379 1946 leader_election.cc:290] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:18.695394 861 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:18.695672 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6d9face4d3c64b8987085cd5b8662db9 (DEFAULT_TABLE table=default.Table_c8619310a8224be58710ffee5aa94c98 [id=4703c4dbffee4b3098c563c6d304c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.696126 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d9face4d3c64b8987085cd5b8662db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.697674 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fcf8a3e0f4770a9743fddffa6495f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.698181 861 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.698279 861 leader_election.cc:304] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.698645 1946 raft_consensus.cc:2749] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.701418 1178 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:18.701617 1178 ts_tablet_manager.cc:1403] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.202s user 0.004s sys 0.003s
I20251029 08:52:18.702448 1178 raft_consensus.cc:359] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.702662 1178 raft_consensus.cc:385] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.702752 1178 raft_consensus.cc:740] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.703028 1178 consensus_queue.cc:260] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.708550 1178 ts_tablet_manager.cc:1434] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:18.709776 1178 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:18.711815 1183 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:18.711995 1183 ts_tablet_manager.cc:1403] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.275s user 0.008s sys 0.004s
I20251029 08:52:18.713150 1183 raft_consensus.cc:359] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.713369 1183 raft_consensus.cc:385] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.713441 1183 raft_consensus.cc:740] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.713670 1183 consensus_queue.cc:260] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.714196 1183 ts_tablet_manager.cc:1434] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:18.714478 1183 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:18.715229 2038 raft_consensus.cc:493] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.715428 2038 raft_consensus.cc:515] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.717801 2038 leader_election.cc:290] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.718917 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aafa846cb2144cec9ce4b28e61d91961" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.719537 718 leader_election.cc:343] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.723898 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aafa846cb2144cec9ce4b28e61d91961" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.724432 720 leader_election.cc:343] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.724592 720 leader_election.cc:304] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.724890 2038 raft_consensus.cc:2749] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.727178 893 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5d96adac7a4fd982d9e574b89c2982 (DEFAULT_TABLE table=default.Table_d7e9b2817baa4c87a5adad8a8905c6ef [id=f738b848eae64e4da114bbc7c2a2588c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.727495 902 tablet_service.cc:1505] Processing CreateTablet for tablet e14d05e64689423da77d6d60b760cc06 (DEFAULT_TABLE table=default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71 [id=89283190678041a089bf5a813de0152d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.727677 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5d96adac7a4fd982d9e574b89c2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.728004 892 tablet_service.cc:1505] Processing CreateTablet for tablet 6d9face4d3c64b8987085cd5b8662db9 (DEFAULT_TABLE table=default.Table_c8619310a8224be58710ffee5aa94c98 [id=4703c4dbffee4b3098c563c6d304c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.728312 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d9face4d3c64b8987085cd5b8662db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.730319 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e4f80728a7a04583bada485707beb806 (table default.Table_18a7eae91e5b4ab595c36f630248860f [id=99ae7a11d37346e18a01a330d7fa42b1]) successfully deleted
I20251029 08:52:18.732234 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 (DEFAULT_TABLE table=default.Table_f40b8aaae0b34603be167b08d72d08d3 [id=452d1fa58caf4ea6b9aed2783fd1caad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.732383 887 tablet_service.cc:1505] Processing CreateTablet for tablet 850aee25ae1940bcbe6677dcea2e7b1d (DEFAULT_TABLE table=default.Table_4f68e8291cd64391bdb236f948dd338b [id=4d90d1e2a9af404a9568241d66028332]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.732645 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e96f14b6c74450828ef50148ab7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.732817 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 850aee25ae1940bcbe6677dcea2e7b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.736423 901 tablet_service.cc:1505] Processing CreateTablet for tablet 61e96f14b6c74450828ef50148ab7ac5 (DEFAULT_TABLE table=default.Table_f40b8aaae0b34603be167b08d72d08d3 [id=452d1fa58caf4ea6b9aed2783fd1caad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.736843 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e96f14b6c74450828ef50148ab7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.738605 590 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc] alter complete (version 2)
I20251029 08:52:18.740567 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d05e64689423da77d6d60b760cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.743414 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9a603e0ef22e48f18263ece15b799c0c
I20251029 08:52:18.743903 597 catalog_manager.cc:6719] ReplaceTablet: tablet 9a603e0ef22e48f18263ece15b799c0c deleted and replaced by tablet 33f1feee1c4d4a98bdb23ac66ac337aa
I20251029 08:52:18.744266 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_82b40b4d97fe4af1bc8a1d95986ab2c9" } new_table_name: "default.Table_3584ce3292564e359247b74ca750e4d9" modify_external_catalogs: true
I20251029 08:52:18.744717 1974 ts_tablet_manager.cc:1929] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.744880 1974 log.cc:1199] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/f33c0bbf104046ccb86ff313f08da22a
I20251029 08:52:18.746225 746 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.746515 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a603e0ef22e48f18263ece15b799c0c with delete_type TABLET_DATA_DELETED (replaced by tablet 33f1feee1c4d4a98bdb23ac66ac337aa) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.746727 1974 ts_tablet_manager.cc:1950] T f33c0bbf104046ccb86ff313f08da22a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.747000 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.747612 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.748127 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 748) took 1128 ms. Trace:
I20251029 08:52:18.749126 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:18.749456 1974 rpcz_store.cc:276] 1029 08:52:17.626973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.631782 (+ 4809us) service_pool.cc:225] Handling call
1029 08:52:18.656175 (+1024393us) ts_tablet_manager.cc:1151] Deleting tablet f33c0bbf104046ccb86ff313f08da22a
1029 08:52:18.656188 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.672832 (+ 16644us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.744631 (+ 71799us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.748081 (+ 3450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024302}
I20251029 08:52:18.749646 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:18.749965 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:18.750296 1974 tablet_replica.cc:333] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.750802 1974 raft_consensus.cc:2243] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.751468 2038 raft_consensus.cc:493] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.751672 2038 raft_consensus.cc:515] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.751833 1974 raft_consensus.cc:2272] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.752502 2038 leader_election.cc:290] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.753006 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de78ea608c904aeea7f7145b9c80f45b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.753615 718 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.754582 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a603e0ef22e48f18263ece15b799c0c with delete_type TABLET_DATA_DELETED (replaced by tablet 33f1feee1c4d4a98bdb23ac66ac337aa) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.754613 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de78ea608c904aeea7f7145b9c80f45b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:18.755462 720 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.755625 720 leader_election.cc:304] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.755940 2038 raft_consensus.cc:2749] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.758594 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a603e0ef22e48f18263ece15b799c0c with delete_type TABLET_DATA_DELETED (replaced by tablet 33f1feee1c4d4a98bdb23ac66ac337aa) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.764583 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet f33c0bbf104046ccb86ff313f08da22a (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
W20251029 08:52:18.767805 996 consensus_peers.cc:597] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2773cae9814df79ee60ea8b0fe4e9d. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:18.768790 1183 tablet_bootstrap.cc:654] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.772395 1974 ts_tablet_manager.cc:1916] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.779009 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:18.780884 595 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_35d49c0bbc6d402a8b18885c966ad1e3 [id=acafbaf717604c2e87a1b9e6263dcf48] alter complete (version 2)
I20251029 08:52:18.782598 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2964a005ddb04ec6aae15abc146d4b82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.783404 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2964a005ddb04ec6aae15abc146d4b82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.787216 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 85df43bf781c421688f163331de0eb2b
I20251029 08:52:18.787984 596 master_service.cc:954] ReplaceTablet: received request to replace tablet c9bbf6401423411089c9070f3b03f1a7 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:18.790518 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 85df43bf781c421688f163331de0eb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.797170 1946 raft_consensus.cc:493] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.797364 1946 raft_consensus.cc:515] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:18.797712 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23dc7808f8da41e3aa0af5f318cd05c1 consensus state has no leader
I20251029 08:52:18.797981 1946 leader_election.cc:290] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:18.798751 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d4b6d93a144541b3b24624df6e5a0201" } new_table_name: "default.Table_bce9df9d85294f4fb7c3b786993463ca" modify_external_catalogs: true
I20251029 08:52:18.798748 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07906300f1d54d4e8fde5165ee14c44f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.799007 1070 raft_consensus.cc:2468] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:18.799873 861 leader_election.cc:304] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:18.800622 1946 raft_consensus.cc:2804] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:18.800724 1946 raft_consensus.cc:493] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:18.800802 1946 raft_consensus.cc:3060] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.801331 753 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.802052 753 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.802698 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:18.803951 861 leader_election.cc:343] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 07906300f1d54d4e8fde5165ee14c44f
I20251029 08:52:18.804597 1946 raft_consensus.cc:515] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.805616 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07906300f1d54d4e8fde5165ee14c44f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:18.805799 1070 raft_consensus.cc:3060] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.808858 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:18.808993 1070 raft_consensus.cc:2468] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:18.809912 861 leader_election.cc:304] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:18.809823 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a3002c1ea00f4683adab3b90c76dde56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:18.810284 2016 raft_consensus.cc:2804] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:18.810391 2016 raft_consensus.cc:697] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
W20251029 08:52:18.810489 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3002c1ea00f4683adab3b90c76dde56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.810638 2016 consensus_queue.cc:237] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.812270 597 catalog_manager.cc:5649] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:18.812856 597 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:18.812958 1946 leader_election.cc:290] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:18.814128 861 leader_election.cc:343] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 07906300f1d54d4e8fde5165ee14c44f
W20251029 08:52:18.814224 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a840d5bb0c374f66aac572ffc08008b3 consensus state has no leader
I20251029 08:52:18.814690 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a10a39719edd4acfb78ffc1e3c43d550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.815317 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a10a39719edd4acfb78ffc1e3c43d550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.816512 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.816793 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 85df43bf781c421688f163331de0eb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.816890 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.817134 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c312fa246a2c4435b6a2c1fa67420ec0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.817709 996 consensus_peers.cc:597] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:18.817858 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c312fa246a2c4435b6a2c1fa67420ec0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.818516 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9d51a88fcb8d479f8b0e2d77e02cf90a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:18.819085 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d51a88fcb8d479f8b0e2d77e02cf90a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:18.820406 594 master_service.cc:954] ReplaceTablet: received request to replace tablet bda3e143699b41b6bba308c83bbb3e0d from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:18.822407 996 consensus_peers.cc:597] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:18.824497 1946 consensus_queue.cc:1048] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:18.837174 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:18.841223 582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 69732.8us
I20251029 08:52:18.843691 1043 tablet_service.cc:1505] Processing CreateTablet for tablet d29f1a5dfc3547f8a555e1956c5ae5b6 (DEFAULT_TABLE table=default.Table_6773a40abf1c451bbba19564445118ab [id=11ef8c91a03a42758d6396bfe1240da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.844388 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29f1a5dfc3547f8a555e1956c5ae5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.848366 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.848619 902 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.848760 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 85df43bf781c421688f163331de0eb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.848795 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 072cf27b0743488c8166e03b5e61c545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.848917 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5385fef404a4463595074f0c8ff822d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.849992 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 07906300f1d54d4e8fde5165ee14c44f with delete_type TABLET_DATA_DELETED (replaced by tablet a92cc4a93a584dffaafe21316fe8dc06) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.864949 896 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:18.870795 896 tablet_service.cc:1505] Processing CreateTablet for tablet d29f1a5dfc3547f8a555e1956c5ae5b6 (DEFAULT_TABLE table=default.Table_6773a40abf1c451bbba19564445118ab [id=11ef8c91a03a42758d6396bfe1240da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.873435 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29f1a5dfc3547f8a555e1956c5ae5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.799885 1974 ts_tablet_manager.cc:1929] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.873845 1974 log.cc:1199] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e5fa181849474a559701fdb8c18d398b
I20251029 08:52:18.875367 1974 ts_tablet_manager.cc:1950] T e5fa181849474a559701fdb8c18d398b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.876577 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 749) took 1249 ms. Trace:
I20251029 08:52:18.878654 748 tablet_service.cc:1505] Processing CreateTablet for tablet d29f1a5dfc3547f8a555e1956c5ae5b6 (DEFAULT_TABLE table=default.Table_6773a40abf1c451bbba19564445118ab [id=11ef8c91a03a42758d6396bfe1240da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.878811 1974 rpcz_store.cc:276] 1029 08:52:17.627163 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.631868 (+ 4705us) service_pool.cc:225] Handling call
1029 08:52:18.750244 (+1118376us) ts_tablet_manager.cc:1151] Deleting tablet e5fa181849474a559701fdb8c18d398b
1029 08:52:18.750255 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.775953 (+ 25698us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.799771 (+ 23818us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.876555 (+ 76784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118294}
I20251029 08:52:18.879176 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29f1a5dfc3547f8a555e1956c5ae5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.865906 1183 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:18.874171 2044 raft_consensus.cc:493] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.880389 2044 raft_consensus.cc:515] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:18.882735 998 leader_election.cc:343] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5385fef404a4463595074f0c8ff822d7
I20251029 08:52:18.883266 2044 leader_election.cc:290] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.884218 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5385fef404a4463595074f0c8ff822d7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:18.884485 922 raft_consensus.cc:2468] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:18.884920 996 leader_election.cc:304] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.884212 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 753) took 1235 ms. Trace:
I20251029 08:52:18.885317 2044 raft_consensus.cc:2804] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:18.885417 2044 raft_consensus.cc:493] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:18.885475 2044 raft_consensus.cc:3060] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.885313 1974 rpcz_store.cc:276] 1029 08:52:17.648566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.651890 (+ 3324us) service_pool.cc:225] Handling call
1029 08:52:18.879989 (+1228099us) ts_tablet_manager.cc:1151] Deleting tablet e4190dc9e09b4480a79774d1e0b5bc2c
1029 08:52:18.880004 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.884194 (+ 4190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227843}
I20251029 08:52:18.886328 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 740) took 1619 ms. Trace:
I20251029 08:52:18.886422 1974 rpcz_store.cc:276] 1029 08:52:17.267276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.629348 (+362072us) service_pool.cc:225] Handling call
1029 08:52:18.886180 (+1256832us) ts_tablet_manager.cc:1151] Deleting tablet 2f4c8c74a7b64d4ba5d9318bf6b23601
1029 08:52:18.886196 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.886308 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229888}
I20251029 08:52:18.887508 1974 tablet_replica.cc:333] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:18.887756 1974 raft_consensus.cc:2243] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:18.887989 1974 pending_rounds.cc:70] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:18.888928 1974 pending_rounds.cc:77] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037634236100608 op_type: NO_OP noop_request { }
I20251029 08:52:18.889212 1974 raft_consensus.cc:2889] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:18.889441 1974 raft_consensus.cc:2272] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.889750 2044 raft_consensus.cc:515] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.890481 1974 ts_tablet_manager.cc:1916] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:18.892405 998 leader_election.cc:343] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5385fef404a4463595074f0c8ff822d7
I20251029 08:52:18.893196 2044 leader_election.cc:290] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.893843 1183 ts_tablet_manager.cc:1403] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.179s user 0.000s sys 0.004s
I20251029 08:52:18.894888 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5385fef404a4463595074f0c8ff822d7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:18.895066 922 raft_consensus.cc:3060] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:18.897105 1183 raft_consensus.cc:359] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.899457 922 raft_consensus.cc:2468] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:18.903939 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6a0b95dd9d0341d697d8709cd5e42be3" } new_table_name: "default.Table_64051e12990148d9aad8a1bd220d49e8" modify_external_catalogs: true
I20251029 08:52:18.905318 2038 raft_consensus.cc:493] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.905589 2038 raft_consensus.cc:515] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.905835 2129 consensus_queue.cc:1048] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:18.906725 2038 leader_election.cc:290] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.907011 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41e435e9dfb34d4f8b0c8f7b702d5a6c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.907588 718 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.918064 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1f5f2cd9e3f14a0a8f3316489f46929a" } modify_external_catalogs: true
I20251029 08:52:18.919463 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1f5f2cd9e3f14a0a8f3316489f46929a" } modify_external_catalogs: true
I20251029 08:52:18.913712 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e5fa181849474a559701fdb8c18d398b (table default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf [id=ca079b1870a64ab69697d5255756c167]) successfully deleted
I20251029 08:52:18.925201 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 697) took 2171 ms. Trace:
I20251029 08:52:18.927855 1176 rpcz_store.cc:276] 1029 08:52:16.753517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.753592 (+ 75us) service_pool.cc:225] Handling call
1029 08:52:18.503045 (+1749453us) ts_tablet_manager.cc:1151] Deleting tablet 840ff254244f43a18679232279e02d1f
1029 08:52:18.503057 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.545329 (+ 42272us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.564935 (+ 19606us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.925133 (+360198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749231}
I20251029 08:52:18.929015 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 698) took 2141 ms. Trace:
I20251029 08:52:18.929198 1176 rpcz_store.cc:276] 1029 08:52:16.787105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.787171 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:18.928835 (+2141664us) ts_tablet_manager.cc:1151] Deleting tablet c1f6ca7d4e534a46a1fba75acd7df39e
1029 08:52:18.928844 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.928948 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2141428}
I20251029 08:52:18.933262 1176 tablet_replica.cc:333] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:18.933557 1176 raft_consensus.cc:2243] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.933774 1176 raft_consensus.cc:2272] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.928679 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 840ff254244f43a18679232279e02d1f (table default.Table_82ec1f06f21147c991f019803452df8c [id=383565dcefdd479288ddfc261cbced6f]) successfully deleted
W20251029 08:52:18.937484 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e4190dc9e09b4480a79774d1e0b5bc2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4190dc9e09b4480a79774d1e0b5bc2c already in progress: creating tablet
W20251029 08:52:18.937726 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 already in progress: creating tablet
W20251029 08:52:18.937779 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c1f6ca7d4e534a46a1fba75acd7df39e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f6ca7d4e534a46a1fba75acd7df39e already in progress: creating tablet
I20251029 08:52:18.939955 1176 ts_tablet_manager.cc:1916] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:18.900465 996 leader_election.cc:304] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:18.944355 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41e435e9dfb34d4f8b0c8f7b702d5a6c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.944567 1183 raft_consensus.cc:385] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:18.944633 1183 raft_consensus.cc:740] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:18.944445 2044 raft_consensus.cc:2804] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:18.944983 720 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.945106 720 leader_election.cc:304] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.945343 2038 raft_consensus.cc:2749] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.947743 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 9493653c2e354458923e9e8dd0da0c60 (DEFAULT_TABLE table=default.Table_8ffc14da79c544d394a4d7fe49b465f9 [id=a2516619cb024ba5bbb0c7fd7751e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.948251 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9493653c2e354458923e9e8dd0da0c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.948921 1183 consensus_queue.cc:260] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.951160 893 tablet_service.cc:1505] Processing CreateTablet for tablet 2497d2ced3024e09968cebd6961c3a90 (DEFAULT_TABLE table=default.Table_e2f8b7294ccd4916b90086595173eb9f [id=0c9012951c214be5bef67c3a8e9e455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.951606 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2497d2ced3024e09968cebd6961c3a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.953086 2038 raft_consensus.cc:493] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:18.953214 2038 raft_consensus.cc:515] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:18.953811 2038 leader_election.cc:290] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:18.955382 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:18.955792 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:18.955857 720 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:18.956302 718 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:18.956276 747 tablet_service.cc:1505] Processing CreateTablet for tablet 9493653c2e354458923e9e8dd0da0c60 (DEFAULT_TABLE table=default.Table_8ffc14da79c544d394a4d7fe49b465f9 [id=a2516619cb024ba5bbb0c7fd7751e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.956413 718 leader_election.cc:304] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:18.956679 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9493653c2e354458923e9e8dd0da0c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.956779 2038 raft_consensus.cc:2749] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:18.961808 893 tablet_service.cc:1505] Processing CreateTablet for tablet 9493653c2e354458923e9e8dd0da0c60 (DEFAULT_TABLE table=default.Table_8ffc14da79c544d394a4d7fe49b465f9 [id=a2516619cb024ba5bbb0c7fd7751e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.962162 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9493653c2e354458923e9e8dd0da0c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:18.964061 757 tablet_service.cc:1505] Processing CreateTablet for tablet 2497d2ced3024e09968cebd6961c3a90 (DEFAULT_TABLE table=default.Table_e2f8b7294ccd4916b90086595173eb9f [id=0c9012951c214be5bef67c3a8e9e455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:18.964464 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2497d2ced3024e09968cebd6961c3a90. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:18.975062 998 consensus_peers.cc:597] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f33c0bbf104046ccb86ff313f08da22a. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:18.976536 2044 raft_consensus.cc:697] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:18.976717 1176 ts_tablet_manager.cc:1929] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:18.978404 2044 consensus_queue.cc:237] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:18.979245 1176 log.cc:1199] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:18.979647 1176 ts_tablet_manager.cc:1950] T bc9c221eaa9a4721ba25761206a13645 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:18.981446 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 699) took 2183 ms. Trace:
I20251029 08:52:18.982241 1176 rpcz_store.cc:276] 1029 08:52:16.797580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.797631 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:18.933216 (+2135585us) ts_tablet_manager.cc:1151] Deleting tablet bc9c221eaa9a4721ba25761206a13645
1029 08:52:18.933225 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.970496 (+ 37271us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.976624 (+ 6128us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.981423 (+ 4799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135401}
I20251029 08:52:18.982841 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bc9c221eaa9a4721ba25761206a13645 (table default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]) successfully deleted
I20251029 08:52:18.985857 1176 tablet_replica.cc:333] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:18.986076 1176 raft_consensus.cc:2243] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:18.986292 1176 raft_consensus.cc:2272] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:18.986442 1974 ts_tablet_manager.cc:1929] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:18.986575 1974 log.cc:1199] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/5a7a673ba4fd4abcbfc3493c38dcc257
I20251029 08:52:18.986958 1974 ts_tablet_manager.cc:1950] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:18.988130 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 750) took 1339 ms. Trace:
I20251029 08:52:18.988587 1974 rpcz_store.cc:276] 1029 08:52:17.648233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.648300 (+ 67us) service_pool.cc:225] Handling call
1029 08:52:18.887439 (+1239139us) ts_tablet_manager.cc:1151] Deleting tablet 5a7a673ba4fd4abcbfc3493c38dcc257
1029 08:52:18.887453 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.938583 (+ 51130us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.986329 (+ 47746us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:18.988111 (+ 1782us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225932}
I20251029 08:52:18.935724 1179 ts_tablet_manager.cc:1929] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:18.990141 1179 log.cc:1199] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a31847928db44bd4b1ebd50c5130065f
I20251029 08:52:18.990621 1179 ts_tablet_manager.cc:1950] T a31847928db44bd4b1ebd50c5130065f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:18.990744 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 755) took 1330 ms. Trace:
I20251029 08:52:18.986956 1176 ts_tablet_manager.cc:1916] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:18.992066 998 consensus_peers.cc:597] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5fa181849474a559701fdb8c18d398b. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:18.996532 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:18.999449 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:19.003293 2044 raft_consensus.cc:493] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.003779 2044 raft_consensus.cc:515] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:19.005573 998 leader_election.cc:343] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:19.006018 2044 leader_election.cc:290] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.007673 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a31847928db44bd4b1ebd50c5130065f (table default.Table_3f1b48395900452b8b149c6aef010592 [id=fd07f1c36584477caf673f91f2a86f58]) successfully deleted
W20251029 08:52:19.009824 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d896959a4c684e1b8b68a36125e751b9 because the tablet was not found. No further retry: Not found: Tablet not found: d896959a4c684e1b8b68a36125e751b9
I20251029 08:52:19.006235 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e626994e96d54c0d913fdedc85fd1155" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:19.010155 1178 tablet_bootstrap.cc:654] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:18.993695 1974 rpcz_store.cc:276] 1029 08:52:17.660283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.664665 (+ 4382us) service_pool.cc:225] Handling call
1029 08:52:18.990614 (+1325949us) ts_tablet_manager.cc:1151] Deleting tablet 400c4991955b47e799df66c05caee6e3
1029 08:52:18.990624 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:18.990728 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320670}
W20251029 08:52:19.010376 996 leader_election.cc:343] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.010502 996 leader_election.cc:304] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.010883 2044 raft_consensus.cc:2749] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.010993 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 751) took 1362 ms. Trace:
I20251029 08:52:19.011374 1183 ts_tablet_manager.cc:1434] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.115s user 0.001s sys 0.001s
I20251029 08:52:19.011929 1183 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.013962 1183 tablet_bootstrap.cc:654] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.015094 1974 rpcz_store.cc:276] 1029 08:52:17.648389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.648838 (+ 449us) service_pool.cc:225] Handling call
1029 08:52:19.010880 (+1362042us) ts_tablet_manager.cc:1151] Deleting tablet f7b89ced286b4a04a28022917b1d0235
1029 08:52:19.010890 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.010975 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331155}
I20251029 08:52:19.015743 1974 tablet_replica.cc:333] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.015738 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 2497d2ced3024e09968cebd6961c3a90 (DEFAULT_TABLE table=default.Table_e2f8b7294ccd4916b90086595173eb9f [id=0c9012951c214be5bef67c3a8e9e455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.016220 1974 raft_consensus.cc:2243] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.016542 1179 tablet_replica.cc:333] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.016692 1974 raft_consensus.cc:2272] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.017627 745 tablet_service.cc:1505] Processing CreateTablet for tablet b61829fcd59d4ad4ab427270b7089d57 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.018134 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61829fcd59d4ad4ab427270b7089d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.018607 1045 tablet_service.cc:1505] Processing CreateTablet for tablet b61829fcd59d4ad4ab427270b7089d57 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.018317 1179 raft_consensus.cc:2243] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.019218 1179 raft_consensus.cc:2272] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.019526 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb (DEFAULT_TABLE table=default.Table_180698a62af24093bee27beaedfc0c18 [id=47a0661cbcdf4ea29aedbbcdef422904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.021124 758 tablet_service.cc:1505] Processing CreateTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb (DEFAULT_TABLE table=default.Table_180698a62af24093bee27beaedfc0c18 [id=47a0661cbcdf4ea29aedbbcdef422904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.021523 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c93076ca44542a5ae19e2c73dba95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.022070 1179 ts_tablet_manager.cc:1916] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.016522 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2497d2ced3024e09968cebd6961c3a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.023661 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c93076ca44542a5ae19e2c73dba95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.025511 1974 ts_tablet_manager.cc:1916] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.028146 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61829fcd59d4ad4ab427270b7089d57. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:19.032395 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c32679cbb1884f09addb0f7c26ac941e consensus state has no leader
I20251029 08:52:19.032852 2046 raft_consensus.cc:493] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.032997 2046 raft_consensus.cc:515] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.033696 2046 leader_election.cc:290] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:19.037261 861 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:19.037909 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c572c68b32d443b2a019e1f4bf2d82fe" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.039206 861 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.039330 861 leader_election.cc:304] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:19.039615 2046 raft_consensus.cc:2749] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.043288 892 tablet_service.cc:1505] Processing CreateTablet for tablet b61829fcd59d4ad4ab427270b7089d57 (DEFAULT_TABLE table=default.Table_ffc61d294dca458aba887c7e9614a720 [id=721ed65a3fce426d837c36d130465459]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.043730 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61829fcd59d4ad4ab427270b7089d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.044117 2044 consensus_queue.cc:1048] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:19.051400 887 tablet_service.cc:1505] Processing CreateTablet for tablet abeb25c1428f40a6b689d03a090bfb7b (DEFAULT_TABLE table=default.Table_d178fa8200d645e19cfaa9177254d5e0 [id=ef1982fe1e9e47de92aafb8f141dc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.051723 1176 ts_tablet_manager.cc:1929] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.051816 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abeb25c1428f40a6b689d03a090bfb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.051828 1176 log.cc:1199] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8991ea78321545be85d5c480841e4751
I20251029 08:52:19.052142 1176 ts_tablet_manager.cc:1950] T 8991ea78321545be85d5c480841e4751 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.052893 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 700) took 2226 ms. Trace:
I20251029 08:52:19.055809 1176 rpcz_store.cc:276] 1029 08:52:16.826040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.826091 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:18.985807 (+2159716us) ts_tablet_manager.cc:1151] Deleting tablet 8991ea78321545be85d5c480841e4751
1029 08:52:18.985818 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.037525 (+ 51707us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.051624 (+ 14099us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.052873 (+ 1249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159536}
I20251029 08:52:19.056402 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8991ea78321545be85d5c480841e4751 (table default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]) successfully deleted
I20251029 08:52:19.056794 1176 tablet_replica.cc:333] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.057008 1176 raft_consensus.cc:2243] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.057197 1176 raft_consensus.cc:2272] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.057793 1176 ts_tablet_manager.cc:1916] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.058974 892 tablet_service.cc:1505] Processing CreateTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb (DEFAULT_TABLE table=default.Table_180698a62af24093bee27beaedfc0c18 [id=47a0661cbcdf4ea29aedbbcdef422904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.059417 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c93076ca44542a5ae19e2c73dba95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.061590 1179 ts_tablet_manager.cc:1929] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.066663 1179 log.cc:1199] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0ff4e625abf14fbd94a8102dfa5fc993
I20251029 08:52:19.067018 1179 ts_tablet_manager.cc:1950] T 0ff4e625abf14fbd94a8102dfa5fc993 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.067816 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 722) took 1034 ms. Trace:
I20251029 08:52:19.067946 1179 rpcz_store.cc:276] 1029 08:52:18.033536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.033856 (+ 320us) service_pool.cc:225] Handling call
1029 08:52:19.016505 (+982649us) ts_tablet_manager.cc:1151] Deleting tablet 0ff4e625abf14fbd94a8102dfa5fc993
1029 08:52:19.016512 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.047706 (+ 31194us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.061460 (+ 13754us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.067792 (+ 6332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1343,"tablet-delete.queue_time_us":982149}
I20251029 08:52:19.068526 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0ff4e625abf14fbd94a8102dfa5fc993 (table default.Table_1de2f068a9d64b31b5211cd9c5791707 [id=353f1e0b5b164f878651738f85e39222]) successfully deleted
I20251029 08:52:19.068809 1179 tablet_replica.cc:333] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.069017 1179 raft_consensus.cc:2243] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.069305 1179 pending_rounds.cc:70] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:19.069376 1179 pending_rounds.cc:77] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037633905618944 op_type: NO_OP noop_request { }
I20251029 08:52:19.069500 1179 raft_consensus.cc:2889] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.069617 1179 raft_consensus.cc:2272] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.070432 1179 ts_tablet_manager.cc:1916] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:19.073632 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 400c4991955b47e799df66c05caee6e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400c4991955b47e799df66c05caee6e3 already in progress: creating tablet
W20251029 08:52:19.074011 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f7b89ced286b4a04a28022917b1d0235 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b89ced286b4a04a28022917b1d0235 already in progress: creating tablet
I20251029 08:52:19.078701 1179 ts_tablet_manager.cc:1929] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.086015 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:19.086405 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:19.086524 1176 ts_tablet_manager.cc:1929] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.086645 1176 log.cc:1199] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ea13e9ea2c11479ea86e3fd7f2a9e59c
I20251029 08:52:19.086982 1176 ts_tablet_manager.cc:1950] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.087824 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 701) took 2261 ms. Trace:
I20251029 08:52:19.087937 1176 rpcz_store.cc:276] 1029 08:52:16.826142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.826300 (+ 158us) service_pool.cc:225] Handling call
1029 08:52:19.056743 (+2230443us) ts_tablet_manager.cc:1151] Deleting tablet ea13e9ea2c11479ea86e3fd7f2a9e59c
1029 08:52:19.056755 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.065027 (+ 8272us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.086401 (+ 21374us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.087801 (+ 1400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230316}
I20251029 08:52:19.088550 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (table default.Table_13d81683d3304abba1837078e6685abd [id=994af8859eb748c4a3ffee07d64d1e33]) successfully deleted
I20251029 08:52:19.089026 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 704) took 2093 ms. Trace:
I20251029 08:52:19.089156 1176 rpcz_store.cc:276] 1029 08:52:16.995638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:16.995713 (+ 75us) service_pool.cc:225] Handling call
1029 08:52:19.088888 (+2093175us) ts_tablet_manager.cc:1151] Deleting tablet 116cc0d155e24653937953eea4f39ee4
1029 08:52:19.088899 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.089007 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092975}
W20251029 08:52:19.089646 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 116cc0d155e24653937953eea4f39ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116cc0d155e24653937953eea4f39ee4 already in progress: creating tablet
I20251029 08:52:19.090037 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 722) took 1964 ms. Trace:
I20251029 08:52:19.090154 1176 rpcz_store.cc:276] 1029 08:52:17.126029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.126081 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:19.089930 (+1963849us) ts_tablet_manager.cc:1151] Deleting tablet 29bb35cd36464dbd96470cc78faa481f
1029 08:52:19.089939 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.090021 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963673}
W20251029 08:52:19.090615 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 29bb35cd36464dbd96470cc78faa481f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29bb35cd36464dbd96470cc78faa481f already in progress: creating tablet
I20251029 08:52:19.091035 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 733) took 1843 ms. Trace:
I20251029 08:52:19.091235 1179 log.cc:1199] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4e2773cae9814df79ee60ea8b0fe4e9d
I20251029 08:52:19.091363 1176 rpcz_store.cc:276] 1029 08:52:17.247308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.247947 (+ 639us) service_pool.cc:225] Handling call
1029 08:52:19.090927 (+1842980us) ts_tablet_manager.cc:1151] Deleting tablet fd4ac3f024a74eb592a1de6fd8f57985
1029 08:52:19.090939 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.091019 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668567}
I20251029 08:52:19.091918 1179 ts_tablet_manager.cc:1950] T 4e2773cae9814df79ee60ea8b0fe4e9d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:19.092304 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fd4ac3f024a74eb592a1de6fd8f57985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd4ac3f024a74eb592a1de6fd8f57985 already in progress: creating tablet
I20251029 08:52:19.092706 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 734) took 1845 ms. Trace:
I20251029 08:52:19.093864 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 723) took 1059 ms. Trace:
I20251029 08:52:19.092828 1176 rpcz_store.cc:276] 1029 08:52:17.247396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.248140 (+ 744us) service_pool.cc:225] Handling call
1029 08:52:19.092593 (+1844453us) ts_tablet_manager.cc:1151] Deleting tablet 98324d4ae1af45fa94ae438dc2a64aaa
1029 08:52:19.092603 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.092690 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669891}
I20251029 08:52:19.094321 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bda3e143699b41b6bba308c83bbb3e0d
W20251029 08:52:19.094450 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 98324d4ae1af45fa94ae438dc2a64aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98324d4ae1af45fa94ae438dc2a64aaa already in progress: creating tablet
I20251029 08:52:19.094755 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 735) took 1847 ms. Trace:
I20251029 08:52:19.094810 594 catalog_manager.cc:6719] ReplaceTablet: tablet bda3e143699b41b6bba308c83bbb3e0d deleted and replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad
I20251029 08:52:19.094828 1176 rpcz_store.cc:276] 1029 08:52:17.247465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.250997 (+ 3532us) service_pool.cc:225] Handling call
1029 08:52:19.094657 (+1843660us) ts_tablet_manager.cc:1151] Deleting tablet 2f4c8c74a7b64d4ba5d9318bf6b23601
1029 08:52:19.094667 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.094740 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1670197}
I20251029 08:52:19.095153 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c0e519f0c5604145a2472a038ef78e46" } modify_external_catalogs: true
I20251029 08:52:19.095367 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c0e519f0c5604145a2472a038ef78e46" } modify_external_catalogs: true
W20251029 08:52:19.097223 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f4c8c74a7b64d4ba5d9318bf6b23601 already in progress: creating tablet
I20251029 08:52:19.098063 758 tablet_service.cc:1552] Processing DeleteTablet for tablet bda3e143699b41b6bba308c83bbb3e0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.098315 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9c7a9b9d1c70476ebfe5556126bb0707" } modify_external_catalogs: true
I20251029 08:52:19.098493 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9c7a9b9d1c70476ebfe5556126bb0707" } modify_external_catalogs: true
I20251029 08:52:19.098845 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 736) took 1851 ms. Trace:
I20251029 08:52:19.098979 1176 rpcz_store.cc:276] 1029 08:52:17.247526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.254179 (+ 6653us) service_pool.cc:225] Handling call
1029 08:52:19.098736 (+1844557us) ts_tablet_manager.cc:1151] Deleting tablet 6689cf0804f946c18ebc555a191549d4
1029 08:52:19.098745 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.098828 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674071}
I20251029 08:52:19.099197 1179 rpcz_store.cc:276] 1029 08:52:18.034140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.034184 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:19.068363 (+1034179us) ts_tablet_manager.cc:1151] Deleting tablet 4e2773cae9814df79ee60ea8b0fe4e9d
1029 08:52:19.068376 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.075641 (+ 7265us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.078612 (+ 2971us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.093842 (+ 15230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1034054}
W20251029 08:52:19.099778 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6689cf0804f946c18ebc555a191549d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6689cf0804f946c18ebc555a191549d4 already in progress: creating tablet
I20251029 08:52:19.100121 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71" } modify_external_catalogs: true
I20251029 08:52:19.100302 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7fbf50cfddb04e4cb8cf9fb87cec1e71" } modify_external_catalogs: true
I20251029 08:52:19.100483 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 737) took 1662 ms. Trace:
I20251029 08:52:19.100560 1176 rpcz_store.cc:276] 1029 08:52:17.437529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.504260 (+ 66731us) service_pool.cc:225] Handling call
1029 08:52:19.100371 (+1596111us) ts_tablet_manager.cc:1151] Deleting tablet d306be1d51674c2abc506d248e05a9c7
1029 08:52:19.100381 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.100464 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595890}
W20251029 08:52:19.100966 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d306be1d51674c2abc506d248e05a9c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d306be1d51674c2abc506d248e05a9c7 already in progress: creating tablet
I20251029 08:52:19.101264 1179 tablet_replica.cc:333] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.102018 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4e2773cae9814df79ee60ea8b0fe4e9d (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:19.102098 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet bda3e143699b41b6bba308c83bbb3e0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.102545 1179 raft_consensus.cc:2243] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.102613 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 663d6587a5e14519afbb0f94d691d1b4 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.102914 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 738) took 1665 ms. Trace:
I20251029 08:52:19.103282 1179 pending_rounds.cc:70] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:19.103404 1176 rpcz_store.cc:276] 1029 08:52:17.437602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.504524 (+ 66922us) service_pool.cc:225] Handling call
1029 08:52:19.102795 (+1598271us) ts_tablet_manager.cc:1151] Deleting tablet f7b89ced286b4a04a28022917b1d0235
1029 08:52:19.102807 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.102896 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598127}
I20251029 08:52:19.103637 1179 pending_rounds.cc:77] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037634045874176 op_type: NO_OP noop_request { }
W20251029 08:52:19.104013 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f7b89ced286b4a04a28022917b1d0235 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b89ced286b4a04a28022917b1d0235 already in progress: creating tablet
I20251029 08:52:19.104418 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 739) took 1565 ms. Trace:
I20251029 08:52:19.104554 1179 raft_consensus.cc:2889] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.104724 1179 raft_consensus.cc:2272] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.105167 1176 rpcz_store.cc:276] 1029 08:52:17.539337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.539410 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:19.104309 (+1564899us) ts_tablet_manager.cc:1151] Deleting tablet e4190dc9e09b4480a79774d1e0b5bc2c
1029 08:52:19.104321 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.104399 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564561}
I20251029 08:52:19.106853 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]). Expected version 2 got 0
I20251029 08:52:19.107177 597 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:19.107143 2143 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:19.108148 2144 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:19.108626 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.109136 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e4190dc9e09b4480a79774d1e0b5bc2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4190dc9e09b4480a79774d1e0b5bc2c already in progress: creating tablet
W20251029 08:52:19.109431 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.109794 1179 ts_tablet_manager.cc:1916] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.110141 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 741) took 1474 ms. Trace:
I20251029 08:52:19.110280 1176 rpcz_store.cc:276] 1029 08:52:17.635270 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.635340 (+ 70us) service_pool.cc:225] Handling call
1029 08:52:19.110018 (+1474678us) ts_tablet_manager.cc:1151] Deleting tablet 400c4991955b47e799df66c05caee6e3
1029 08:52:19.110028 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.110122 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438172}
W20251029 08:52:19.110783 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 400c4991955b47e799df66c05caee6e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400c4991955b47e799df66c05caee6e3 already in progress: creating tablet
I20251029 08:52:19.111116 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b2f1e53945774829a74b8125ee64a77a" } new_table_name: "default.Table_47b07a98fcc34ec2a2784a43b313f904" modify_external_catalogs: true
I20251029 08:52:19.113173 2044 raft_consensus.cc:493] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.113291 1176 tablet_replica.cc:333] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.113394 2044 raft_consensus.cc:515] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.114066 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:19.114535 2044 leader_election.cc:290] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.114795 1176 raft_consensus.cc:2243] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.114962 1176 raft_consensus.cc:2272] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.115047 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc9c221eaa9a4721ba25761206a13645" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:19.115504 1176 ts_tablet_manager.cc:1916] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.115990 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_44386c352f5e4b9795c8df047079ba09" } modify_external_catalogs: true
W20251029 08:52:19.114843 998 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:19.116248 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_44386c352f5e4b9795c8df047079ba09" } modify_external_catalogs: true
W20251029 08:52:19.116510 996 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.116659 996 leader_election.cc:304] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.117195 2044 raft_consensus.cc:2749] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:19.124112 1177 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.063s sys 0.095s
W20251029 08:52:19.124217 1177 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.543s user 0.063s sys 0.095s
I20251029 08:52:19.125001 1177 tablet_bootstrap.cc:492] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.125008 1178 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.125129 1177 ts_tablet_manager.cc:1403] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.550s user 0.065s sys 0.095s
I20251029 08:52:19.125138 1178 ts_tablet_manager.cc:1403] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.416s user 0.001s sys 0.002s
I20251029 08:52:19.125799 1177 raft_consensus.cc:359] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.125967 1177 raft_consensus.cc:385] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.126027 1177 raft_consensus.cc:740] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.125936 1178 raft_consensus.cc:359] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.126132 1178 raft_consensus.cc:385] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.126210 1177 consensus_queue.cc:260] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.126272 1178 raft_consensus.cc:740] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.126693 1177 ts_tablet_manager.cc:1434] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:19.126725 1178 consensus_queue.cc:260] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.127449 1178 ts_tablet_manager.cc:1434] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:19.127849 1178 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.128345 1177 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.129745 1177 tablet_bootstrap.cc:654] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.133977 893 tablet_service.cc:1552] Processing DeleteTablet for tablet bda3e143699b41b6bba308c83bbb3e0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.134500 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.139271 1176 ts_tablet_manager.cc:1929] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.141896 1179 ts_tablet_manager.cc:1929] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.142138 1179 log.cc:1199] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e5fa181849474a559701fdb8c18d398b
I20251029 08:52:19.142596 1179 ts_tablet_manager.cc:1950] T e5fa181849474a559701fdb8c18d398b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.143464 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 724) took 1109 ms. Trace:
I20251029 08:52:19.146917 758 tablet_service.cc:1505] Processing CreateTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.147737 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f58be6d7f5492395ca6192f7ce0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.148917 747 tablet_service.cc:1505] Processing CreateTablet for tablet a08630cbc401467a9dde1ce1e198cef7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.149230 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08630cbc401467a9dde1ce1e198cef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.154094 893 tablet_service.cc:1505] Processing CreateTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.154697 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f58be6d7f5492395ca6192f7ce0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.156107 1176 log.cc:1199] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d896959a4c684e1b8b68a36125e751b9
I20251029 08:52:19.156559 1176 ts_tablet_manager.cc:1950] T d896959a4c684e1b8b68a36125e751b9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.156876 2134 consensus_queue.cc:1048] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:19.152369 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 (DEFAULT_TABLE table=default.Table_44386c352f5e4b9795c8df047079ba09 [id=61f6ac1a72e646d6981bef3ad7a06a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.157567 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f58be6d7f5492395ca6192f7ce0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.157796 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 742) took 1419 ms. Trace:
I20251029 08:52:19.158133 1176 rpcz_store.cc:276] 1029 08:52:17.738498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.739059 (+ 561us) service_pool.cc:225] Handling call
1029 08:52:19.113233 (+1374174us) ts_tablet_manager.cc:1151] Deleting tablet d896959a4c684e1b8b68a36125e751b9
1029 08:52:19.113245 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.131241 (+ 17996us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.139152 (+ 7911us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.157776 (+ 18624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362233}
I20251029 08:52:19.158707 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d896959a4c684e1b8b68a36125e751b9 (table default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]) successfully deleted
I20251029 08:52:19.159091 1176 tablet_replica.cc:333] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.159323 1176 raft_consensus.cc:2243] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.159528 1176 raft_consensus.cc:2272] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.152720 1043 tablet_service.cc:1505] Processing CreateTablet for tablet abeb25c1428f40a6b689d03a090bfb7b (DEFAULT_TABLE table=default.Table_d178fa8200d645e19cfaa9177254d5e0 [id=ef1982fe1e9e47de92aafb8f141dc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.160161 1176 ts_tablet_manager.cc:1916] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.160418 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abeb25c1428f40a6b689d03a090bfb7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:19.168800 996 consensus_peers.cc:597] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f33c0bbf104046ccb86ff313f08da22a. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:19.156790 1179 rpcz_store.cc:276] 1029 08:52:18.034482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.034528 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:19.101216 (+1066688us) ts_tablet_manager.cc:1151] Deleting tablet e5fa181849474a559701fdb8c18d398b
1029 08:52:19.101225 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.121397 (+ 20172us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.124505 (+ 3108us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.143442 (+ 18937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066521}
I20251029 08:52:19.176501 1176 ts_tablet_manager.cc:1929] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.176839 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e5fa181849474a559701fdb8c18d398b (table default.Table_520f44fe63ab4ef49daa9f0e1e89ffcf [id=ca079b1870a64ab69697d5255756c167]) successfully deleted
I20251029 08:52:19.177629 1176 log.cc:1199] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:19.177668 757 tablet_service.cc:1505] Processing CreateTablet for tablet abeb25c1428f40a6b689d03a090bfb7b (DEFAULT_TABLE table=default.Table_d178fa8200d645e19cfaa9177254d5e0 [id=ef1982fe1e9e47de92aafb8f141dc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.178227 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abeb25c1428f40a6b689d03a090bfb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.179020 1176 ts_tablet_manager.cc:1950] T 2f40aafe7dc94851a67abd84d80d7d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.179865 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 744) took 1416 ms. Trace:
I20251029 08:52:19.180467 1179 tablet_replica.cc:333] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.180644 1176 rpcz_store.cc:276] 1029 08:52:17.762936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.762987 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:19.159037 (+1396050us) ts_tablet_manager.cc:1151] Deleting tablet 2f40aafe7dc94851a67abd84d80d7d7b
1029 08:52:19.159048 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.173157 (+ 14109us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.176388 (+ 3231us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.179841 (+ 3453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395853}
I20251029 08:52:19.180774 1179 raft_consensus.cc:2243] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.181190 1179 pending_rounds.cc:70] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:19.181258 1179 pending_rounds.cc:77] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037633896513536 op_type: NO_OP noop_request { }
I20251029 08:52:19.181391 1179 raft_consensus.cc:2889] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.181491 1179 raft_consensus.cc:2272] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.181766 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2f40aafe7dc94851a67abd84d80d7d7b (table default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]) successfully deleted
I20251029 08:52:19.182183 1176 tablet_replica.cc:333] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.183349 1176 raft_consensus.cc:2243] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.183602 1176 raft_consensus.cc:2272] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.184170 2016 raft_consensus.cc:493] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.184394 2016 raft_consensus.cc:515] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.186110 1176 ts_tablet_manager.cc:1916] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.188249 896 tablet_service.cc:1505] Processing CreateTablet for tablet a08630cbc401467a9dde1ce1e198cef7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.188707 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08630cbc401467a9dde1ce1e198cef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.188721 2016 leader_election.cc:290] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.189252 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:19.189638 780 raft_consensus.cc:2468] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:19.185657 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.188139 1045 tablet_service.cc:1505] Processing CreateTablet for tablet a08630cbc401467a9dde1ce1e198cef7 (DEFAULT_TABLE table=default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:19.198784 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d292c29d2b34086a5976da251f9ed5e consensus state has no leader
I20251029 08:52:19.198823 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08630cbc401467a9dde1ce1e198cef7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:19.200459 861 leader_election.cc:343] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.202744 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:19.205919 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:19.208662 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:19.206182 1176 ts_tablet_manager.cc:1929] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.209738 1176 log.cc:1199] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:19.210141 1176 ts_tablet_manager.cc:1950] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.210127 861 leader_election.cc:304] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:19.210744 2046 raft_consensus.cc:2804] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:19.210847 2046 raft_consensus.cc:493] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:19.210930 2046 raft_consensus.cc:3060] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.209153 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cf09e8fd3d2b43318fa37d922c8b826f
I20251029 08:52:19.212615 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9fdee9ea81294ee1806bada6c7a10d38" } new_table_name: "default.Table_a6e64c007b434fc3941ebfbd007286b4" modify_external_catalogs: true
I20251029 08:52:19.212882 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 745) took 1449 ms. Trace:
I20251029 08:52:19.212972 1176 rpcz_store.cc:276] 1029 08:52:17.763368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.763411 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:19.182129 (+1418718us) ts_tablet_manager.cc:1151] Deleting tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc
1029 08:52:19.182139 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.199331 (+ 17192us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.206071 (+ 6740us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.212853 (+ 6782us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"tablet-delete.queue_time_us":1418582}
I20251029 08:52:19.214229 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (table default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]) successfully deleted
I20251029 08:52:19.209442 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 62b7fe26c7ad48cdb935e8dc6b1962a7
I20251029 08:52:19.214818 2046 raft_consensus.cc:515] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.215034 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0492a662ebb94c54bc22d21a5b46c918" } new_table_name: "default.Table_962836b469ba4947acfc9e22d06e89e0" modify_external_catalogs: true
I20251029 08:52:19.209679 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c4e1b6b6359b437aa295b473fa42dce1
I20251029 08:52:19.216545 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:19.217007 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.217262 748 tablet_service.cc:1552] Processing DeleteTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.217607 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ea4a80c608ad451aacff1c9c84ae64e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.217748 861 leader_election.cc:343] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:19.218225 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea4a80c608ad451aacff1c9c84ae64e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.218636 2046 leader_election.cc:290] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.218724 782 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:19.218919 782 raft_consensus.cc:3060] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.219203 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.220296 1179 ts_tablet_manager.cc:1916] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.221778 782 raft_consensus.cc:2468] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:19.222143 1176 tablet_replica.cc:333] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.222474 1176 raft_consensus.cc:2243] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.222580 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.222657 1176 raft_consensus.cc:2272] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.222678 861 leader_election.cc:304] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:19.223194 1176 ts_tablet_manager.cc:1916] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.223271 2046 raft_consensus.cc:2804] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:19.223414 2046 raft_consensus.cc:697] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:19.223747 2046 consensus_queue.cc:237] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.220542 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.226035 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:19.227300 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.228899 2038 raft_consensus.cc:493] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.229024 2038 raft_consensus.cc:515] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.229197 1183 tablet_bootstrap.cc:492] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.229364 1183 ts_tablet_manager.cc:1403] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.218s user 0.003s sys 0.000s
I20251029 08:52:19.229640 2038 leader_election.cc:290] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.230378 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e1f0cdfbc34c6f85f2ad545eb33c0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.230537 1183 raft_consensus.cc:359] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.230700 1183 raft_consensus.cc:385] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.230762 1183 raft_consensus.cc:740] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:19.231021 720 leader_election.cc:343] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.231005 1183 consensus_queue.cc:260] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.231098 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e1f0cdfbc34c6f85f2ad545eb33c0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:19.231799 718 leader_election.cc:343] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.231942 718 leader_election.cc:304] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.232272 2038 raft_consensus.cc:2749] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.232646 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bbcfd7574bc64702be1c06a2105814d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.233458 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbcfd7574bc64702be1c06a2105814d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.233781 2148 tablet.cc:1722] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 08:52:19.235102 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c3e50018f414f4b88e0e5d90cb7c925 with delete_type TABLET_DATA_DELETED (replaced by tablet cd6208b92fe94dc881cf45b4627d68f2) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.235769 887 tablet_service.cc:1552] Processing DeleteTablet for tablet cf09e8fd3d2b43318fa37d922c8b826f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.236656 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.237151 887 tablet_service.cc:1552] Processing DeleteTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.237826 1183 ts_tablet_manager.cc:1434] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 08:52:19.239166 1183 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.241137 1183 tablet_bootstrap.cc:654] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.247535 748 tablet_service.cc:1552] Processing DeleteTablet for tablet c4e1b6b6359b437aa295b473fa42dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.250813 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e14d05e64689423da77d6d60b760cc06
I20251029 08:52:19.252830 593 catalog_manager.cc:5649] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:19.253230 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:19.254943 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d05e64689423da77d6d60b760cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.255409 748 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d05e64689423da77d6d60b760cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.256093 1176 ts_tablet_manager.cc:1929] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.256270 1176 log.cc:1199] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:19.256693 1176 ts_tablet_manager.cc:1950] T b7da7e08b74f4a9aacedada42e839a11 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.257529 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 743) took 1494 ms. Trace:
I20251029 08:52:19.260294 892 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d05e64689423da77d6d60b760cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.260886 1176 rpcz_store.cc:276] 1029 08:52:17.762819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.763671 (+ 852us) service_pool.cc:225] Handling call
1029 08:52:19.213410 (+1449739us) ts_tablet_manager.cc:1151] Deleting tablet b7da7e08b74f4a9aacedada42e839a11
1029 08:52:19.222085 (+ 8675us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.228669 (+ 6584us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.255940 (+ 27271us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.257506 (+ 1566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449578}
I20251029 08:52:19.261559 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b7da7e08b74f4a9aacedada42e839a11 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:19.263736 1176 tablet_replica.cc:333] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.266746 1176 raft_consensus.cc:2243] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.267011 1176 raft_consensus.cc:2272] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.267643 1176 ts_tablet_manager.cc:1916] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.270619 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 663d6587a5e14519afbb0f94d691d1b4
I20251029 08:52:19.271139 588 catalog_manager.cc:6719] ReplaceTablet: tablet 663d6587a5e14519afbb0f94d691d1b4 deleted and replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd
I20251029 08:52:19.271581 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4b68078657014c3790ceed68233e826e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.272181 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b68078657014c3790ceed68233e826e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.274178 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:19.275415 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbf9e36c47be4509a998be8e7138475b consensus state has no leader
I20251029 08:52:19.275709 595 master_service.cc:954] ReplaceTablet: received request to replace tablet bbc4345cabbc476485b6a040a076ddaf from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.276965 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d56c826be1cc4fc1a9a8c42bf71799b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.278136 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d56c826be1cc4fc1a9a8c42bf71799b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.280169 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c64ecb648f8c411fa562430e00d12375" } new_table_name: "default.Table_f3ebeac47874490abc642f878752012d" modify_external_catalogs: true
I20251029 08:52:19.283087 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.289465 1974 ts_tablet_manager.cc:1929] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.289963 1045 tablet_service.cc:1505] Processing CreateTablet for tablet ba809e06064141789e7abe6fd3b0567f (DEFAULT_TABLE table=default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1 [id=aa82072873bd4d88b29db475fdc97b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.290377 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba809e06064141789e7abe6fd3b0567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.290457 748 tablet_service.cc:1505] Processing CreateTablet for tablet ba809e06064141789e7abe6fd3b0567f (DEFAULT_TABLE table=default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1 [id=aa82072873bd4d88b29db475fdc97b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.291396 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba809e06064141789e7abe6fd3b0567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.292354 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 7fa387f6096246e68d691c79a93aac64 (DEFAULT_TABLE table=default.Table_fadb0daa1e404b088a08bc134f4bf4a6 [id=839fec3125844dd1af0f2128c710dec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.292702 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa387f6096246e68d691c79a93aac64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.297041 1177 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.297253 1974 log.cc:1199] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d896959a4c684e1b8b68a36125e751b9
I20251029 08:52:19.297312 1177 ts_tablet_manager.cc:1403] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.169s user 0.048s sys 0.073s
I20251029 08:52:19.297580 1974 ts_tablet_manager.cc:1950] T d896959a4c684e1b8b68a36125e751b9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.298091 1177 raft_consensus.cc:359] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.298243 1177 raft_consensus.cc:385] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.298288 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 756) took 1563 ms. Trace:
I20251029 08:52:19.302394 1178 tablet_bootstrap.cc:654] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.298292 1177 raft_consensus.cc:740] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.301179 892 tablet_service.cc:1505] Processing CreateTablet for tablet aed91064bbe8411098d3c1daad5c65cd (DEFAULT_TABLE table=default.Table_f323cb89e8a7460991eb7cfc5da8b754 [id=004075f994c141888ca3aadbf2c9ebdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.302529 1974 rpcz_store.cc:276] 1029 08:52:17.735352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.735411 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:19.015693 (+1280282us) ts_tablet_manager.cc:1151] Deleting tablet d896959a4c684e1b8b68a36125e751b9
1029 08:52:19.015703 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.192218 (+176515us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.289345 (+ 97127us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.298271 (+ 8926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280038}
I20251029 08:52:19.302700 1177 consensus_queue.cc:260] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.302848 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed91064bbe8411098d3c1daad5c65cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.303253 1177 ts_tablet_manager.cc:1434] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:19.303658 1177 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.304113 1036 tablet_service.cc:1505] Processing CreateTablet for tablet aed91064bbe8411098d3c1daad5c65cd (DEFAULT_TABLE table=default.Table_f323cb89e8a7460991eb7cfc5da8b754 [id=004075f994c141888ca3aadbf2c9ebdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.301391 887 tablet_service.cc:1505] Processing CreateTablet for tablet ba809e06064141789e7abe6fd3b0567f (DEFAULT_TABLE table=default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1 [id=aa82072873bd4d88b29db475fdc97b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.304467 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba809e06064141789e7abe6fd3b0567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.304468 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed91064bbe8411098d3c1daad5c65cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.304944 1177 tablet_bootstrap.cc:654] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.302011 901 tablet_service.cc:1505] Processing CreateTablet for tablet 7fa387f6096246e68d691c79a93aac64 (DEFAULT_TABLE table=default.Table_fadb0daa1e404b088a08bc134f4bf4a6 [id=839fec3125844dd1af0f2128c710dec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.308262 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa387f6096246e68d691c79a93aac64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.310267 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d896959a4c684e1b8b68a36125e751b9 (table default.Table_9cd744dfdd12453386d05dbdb493b993 [id=0b6746198ab143dbbd4b4ff86148eac3]) successfully deleted
I20251029 08:52:19.310534 1179 ts_tablet_manager.cc:1929] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.310672 1179 log.cc:1199] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/f33c0bbf104046ccb86ff313f08da22a
I20251029 08:52:19.311012 1179 ts_tablet_manager.cc:1950] T f33c0bbf104046ccb86ff313f08da22a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.311800 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 725) took 1227 ms. Trace:
I20251029 08:52:19.311877 1179 rpcz_store.cc:276] 1029 08:52:18.084294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.084356 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:19.180380 (+1096024us) ts_tablet_manager.cc:1151] Deleting tablet f33c0bbf104046ccb86ff313f08da22a
1029 08:52:19.180399 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.254585 (+ 74186us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.283973 (+ 29388us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.311778 (+ 27805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095619}
I20251029 08:52:19.312296 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet f33c0bbf104046ccb86ff313f08da22a (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
I20251029 08:52:19.312463 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 757) took 1516 ms. Trace:
I20251029 08:52:19.312541 1974 rpcz_store.cc:276] 1029 08:52:17.796100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.796309 (+ 209us) service_pool.cc:225] Handling call
1029 08:52:19.312320 (+1516011us) ts_tablet_manager.cc:1151] Deleting tablet 0bd490da0b0e47fc965343c123a27b22
1029 08:52:19.312333 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.312443 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515821}
W20251029 08:52:19.313007 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0bd490da0b0e47fc965343c123a27b22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bd490da0b0e47fc965343c123a27b22 already in progress: creating tablet
I20251029 08:52:19.313414 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 759) took 1457 ms. Trace:
I20251029 08:52:19.313501 1974 rpcz_store.cc:276] 1029 08:52:17.855541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.855892 (+ 351us) service_pool.cc:225] Handling call
1029 08:52:19.313265 (+1457373us) ts_tablet_manager.cc:1151] Deleting tablet bbc4345cabbc476485b6a040a076ddaf
1029 08:52:19.313276 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.313395 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457189}
W20251029 08:52:19.313897 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bbc4345cabbc476485b6a040a076ddaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc4345cabbc476485b6a040a076ddaf already in progress: creating tablet
I20251029 08:52:19.314150 1974 tablet_replica.cc:333] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.314332 1974 raft_consensus.cc:2243] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.314489 1974 raft_consensus.cc:2272] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.315043 1974 ts_tablet_manager.cc:1916] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.313506 2046 raft_consensus.cc:493] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.316172 2046 raft_consensus.cc:515] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.316700 2046 leader_election.cc:290] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:19.317091 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d30183814f5e4f2ab9d71fe493eb801c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.317859 861 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.319175 1179 tablet_replica.cc:333] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.319406 1179 raft_consensus.cc:2243] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.319936 2038 raft_consensus.cc:493] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.320111 2038 raft_consensus.cc:515] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.320721 2038 leader_election.cc:290] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.321201 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ab9df6f768410eb4f66cffdcd68df2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:19.321705 718 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.326152 747 tablet_service.cc:1505] Processing CreateTablet for tablet aed91064bbe8411098d3c1daad5c65cd (DEFAULT_TABLE table=default.Table_f323cb89e8a7460991eb7cfc5da8b754 [id=004075f994c141888ca3aadbf2c9ebdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.326690 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed91064bbe8411098d3c1daad5c65cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:19.327942 861 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:19.328111 861 leader_election.cc:304] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:19.328338 2046 raft_consensus.cc:2749] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.280275 1176 ts_tablet_manager.cc:1929] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.329857 1176 log.cc:1199] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a3ba512bd4f04bd9aceb4cb210fe9689
I20251029 08:52:19.330209 1176 ts_tablet_manager.cc:1950] T a3ba512bd4f04bd9aceb4cb210fe9689 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.330996 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 747) took 1564 ms. Trace:
I20251029 08:52:19.331139 1176 rpcz_store.cc:276] 1029 08:52:17.766872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.766909 (+ 37us) service_pool.cc:225] Handling call
1029 08:52:19.263684 (+1496775us) ts_tablet_manager.cc:1151] Deleting tablet a3ba512bd4f04bd9aceb4cb210fe9689
1029 08:52:19.263696 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.271053 (+ 7357us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.280151 (+ 9098us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.330977 (+ 50826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496640}
I20251029 08:52:19.321316 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ab9df6f768410eb4f66cffdcd68df2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.332116 861 consensus_peers.cc:597] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 072cf27b0743488c8166e03b5e61c545. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:19.332432 1179 raft_consensus.cc:2272] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:19.332576 720 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.332697 720 leader_election.cc:304] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.333046 2038 raft_consensus.cc:2749] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.333705 1176 tablet_replica.cc:333] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.333673 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a3ba512bd4f04bd9aceb4cb210fe9689 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:19.336210 1176 raft_consensus.cc:2243] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.336436 1176 raft_consensus.cc:2272] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.337041 1176 ts_tablet_manager.cc:1916] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.339830 1183 tablet_bootstrap.cc:492] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.340052 1183 ts_tablet_manager.cc:1403] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.003s
I20251029 08:52:19.341091 1183 raft_consensus.cc:359] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.341368 1183 raft_consensus.cc:385] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.341461 1183 raft_consensus.cc:740] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.341738 1183 consensus_queue.cc:260] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.343617 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7fa387f6096246e68d691c79a93aac64 (DEFAULT_TABLE table=default.Table_fadb0daa1e404b088a08bc134f4bf4a6 [id=839fec3125844dd1af0f2128c710dec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.344189 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa387f6096246e68d691c79a93aac64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.347543 1974 ts_tablet_manager.cc:1929] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.348788 1177 tablet_bootstrap.cc:492] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.353796 1177 ts_tablet_manager.cc:1403] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20251029 08:52:19.354779 887 tablet_service.cc:1505] Processing CreateTablet for tablet 6f525da39ffa43c19838a0c039e4f96b (DEFAULT_TABLE table=default.Table_693b1227de8f476d97871a176deda38f [id=0ee770c0cc204c0aa7631288bca8aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.355154 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f525da39ffa43c19838a0c039e4f96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.355219 1974 log.cc:1199] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/2c706343d44a4807b1ea60c1b1e1ea49
I20251029 08:52:19.355532 1974 ts_tablet_manager.cc:1950] T 2c706343d44a4807b1ea60c1b1e1ea49 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.355830 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 760) took 1437 ms. Trace:
I20251029 08:52:19.355897 1974 rpcz_store.cc:276] 1029 08:52:17.918860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.919126 (+ 266us) service_pool.cc:225] Handling call
1029 08:52:19.314110 (+1394984us) ts_tablet_manager.cc:1151] Deleting tablet 2c706343d44a4807b1ea60c1b1e1ea49
1029 08:52:19.314119 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.344282 (+ 30163us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.347453 (+ 3171us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.355815 (+ 8362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394782}
I20251029 08:52:19.356680 1974 tablet_replica.cc:333] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.357250 1176 ts_tablet_manager.cc:1929] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.357514 1974 raft_consensus.cc:2243] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.357724 1974 raft_consensus.cc:2272] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.358266 1974 ts_tablet_manager.cc:1916] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.358417 1176 log.cc:1199] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/1ae561a995c94fadb288b6395c55a1bf
I20251029 08:52:19.358775 1176 ts_tablet_manager.cc:1950] T 1ae561a995c94fadb288b6395c55a1bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.355165 1177 raft_consensus.cc:359] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.360255 1177 raft_consensus.cc:385] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.360319 1177 raft_consensus.cc:740] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.360556 1177 consensus_queue.cc:260] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.361038 1177 ts_tablet_manager.cc:1434] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:19.361346 1177 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.362311 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 748) took 1595 ms. Trace:
I20251029 08:52:19.363366 1177 tablet_bootstrap.cc:654] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.366058 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 2c706343d44a4807b1ea60c1b1e1ea49 (table default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]) successfully deleted
I20251029 08:52:19.370672 2016 consensus_queue.cc:1048] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:19.370754 1176 rpcz_store.cc:276] 1029 08:52:17.767126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.767162 (+ 36us) service_pool.cc:225] Handling call
1029 08:52:19.333654 (+1566492us) ts_tablet_manager.cc:1151] Deleting tablet 1ae561a995c94fadb288b6395c55a1bf
1029 08:52:19.333666 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.340618 (+ 6952us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.357151 (+ 16533us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.362290 (+ 5139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566378}
I20251029 08:52:19.371114 1176 tablet_replica.cc:333] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.371232 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 1ae561a995c94fadb288b6395c55a1bf (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:19.371307 1176 raft_consensus.cc:2243] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.371455 1176 raft_consensus.cc:2272] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.371968 1176 ts_tablet_manager.cc:1916] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.373694 893 tablet_service.cc:1505] Processing CreateTablet for tablet 491fb831cb2d4dbe9908fb6a571a43ca (DEFAULT_TABLE table=default.Table_22ad2d235d9c4ccdb70814d03790c887 [id=1daa44a8ad23428e9feb291be21d1eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.349555 1179 ts_tablet_manager.cc:1916] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.378569 1183 ts_tablet_manager.cc:1434] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251029 08:52:19.378528 757 tablet_service.cc:1505] Processing CreateTablet for tablet 6f525da39ffa43c19838a0c039e4f96b (DEFAULT_TABLE table=default.Table_693b1227de8f476d97871a176deda38f [id=0ee770c0cc204c0aa7631288bca8aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.366595 748 tablet_service.cc:1505] Processing CreateTablet for tablet 491fb831cb2d4dbe9908fb6a571a43ca (DEFAULT_TABLE table=default.Table_22ad2d235d9c4ccdb70814d03790c887 [id=1daa44a8ad23428e9feb291be21d1eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.370683 1178 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.378868 1178 ts_tablet_manager.cc:1403] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.251s user 0.002s sys 0.002s
I20251029 08:52:19.378934 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f525da39ffa43c19838a0c039e4f96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.379743 1178 raft_consensus.cc:359] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.379922 1178 raft_consensus.cc:385] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.379977 1178 raft_consensus.cc:740] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.380203 1178 consensus_queue.cc:260] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.380740 887 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52c35a09fd4b69b160ecaa2365ddd5 (DEFAULT_TABLE table=default.Table_d55bedf72b97413a8b66d1ab9a7ee591 [id=5ef1cde4a7104193982748c4e181f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.381485 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491fb831cb2d4dbe9908fb6a571a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.386363 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 47f58be6d7f5492395ca6192f7ce0930
I20251029 08:52:19.386963 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491fb831cb2d4dbe9908fb6a571a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.388350 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.390424 1178 ts_tablet_manager.cc:1434] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:19.390774 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52c35a09fd4b69b160ecaa2365ddd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.391678 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.397616 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411] alter complete (version 4)
I20251029 08:52:19.397981 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 4e3a2fab467348099a4a1fec74eb5e65 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.398409 1974 ts_tablet_manager.cc:1929] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.398514 1974 log.cc:1199] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:19.398661 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7c57b7890d1f4c8e8e9801e845e4df11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:19.398856 1974 ts_tablet_manager.cc:1950] T 608b7095d42941ccb0c4eb57f6754e74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.399151 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 761) took 1480 ms. Trace:
I20251029 08:52:19.399219 1974 rpcz_store.cc:276] 1029 08:52:17.918994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.925073 (+ 6079us) service_pool.cc:225] Handling call
1029 08:52:19.356634 (+1431561us) ts_tablet_manager.cc:1151] Deleting tablet 608b7095d42941ccb0c4eb57f6754e74
1029 08:52:19.356644 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.380525 (+ 23881us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.398308 (+ 17783us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.399135 (+ 827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1431384}
W20251029 08:52:19.399317 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c57b7890d1f4c8e8e9801e845e4df11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:19.400308 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccf53a53fbf047329fd95f4deb752991 consensus state has no leader
I20251029 08:52:19.400648 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_45fb9d22c4e64c48b5c2b1312ca0b72e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.401175 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45fb9d22c4e64c48b5c2b1312ca0b72e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:19.402324 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ed4f048f1ae4d70aa1b88116a57e2bd has no consensus state
I20251029 08:52:19.402648 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f29176d94379462f980b648a202d8a6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:19.402842 1178 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:19.403108 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f29176d94379462f980b648a202d8a6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.404568 1178 tablet_bootstrap.cc:654] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.409718 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 608b7095d42941ccb0c4eb57f6754e74 (table default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]) successfully deleted
I20251029 08:52:19.411116 1176 ts_tablet_manager.cc:1929] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.411829 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 762) took 1373 ms. Trace:
I20251029 08:52:19.411959 1974 rpcz_store.cc:276] 1029 08:52:18.038637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.038696 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:19.411683 (+1372987us) ts_tablet_manager.cc:1151] Deleting tablet 4200db822deb4ace936f561c230ac1fb
1029 08:52:19.411695 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.411806 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360840}
I20251029 08:52:19.412500 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 763) took 1356 ms. Trace:
I20251029 08:52:19.412618 1974 rpcz_store.cc:276] 1029 08:52:18.055717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.055771 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:19.412399 (+1356628us) ts_tablet_manager.cc:1151] Deleting tablet 4e3a2fab467348099a4a1fec74eb5e65
1029 08:52:19.412408 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.412484 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356461}
I20251029 08:52:19.413044 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 764) took 1347 ms. Trace:
I20251029 08:52:19.413136 1974 rpcz_store.cc:276] 1029 08:52:18.065653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.065716 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:19.412954 (+1347238us) ts_tablet_manager.cc:1151] Deleting tablet cf7604670833495c9dd61258a3301bb4
1029 08:52:19.412962 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.413029 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347046}
I20251029 08:52:19.413609 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 765) took 1005 ms. Trace:
I20251029 08:52:19.413758 1176 log.cc:1199] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/351047c725f8488abbd074bd40790b48
I20251029 08:52:19.414100 1176 ts_tablet_manager.cc:1950] T 351047c725f8488abbd074bd40790b48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.414206 1974 rpcz_store.cc:276] 1029 08:52:18.408553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.408624 (+ 71us) service_pool.cc:225] Handling call
1029 08:52:19.413496 (+1004872us) ts_tablet_manager.cc:1151] Deleting tablet 084a659932ec4996a6d6b160d67efdd0
1029 08:52:19.413506 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.413591 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":994558}
I20251029 08:52:19.417502 748 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.419028 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 749) took 1651 ms. Trace:
I20251029 08:52:19.419126 1176 rpcz_store.cc:276] 1029 08:52:17.767396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.767431 (+ 35us) service_pool.cc:225] Handling call
1029 08:52:19.371064 (+1603633us) ts_tablet_manager.cc:1151] Deleting tablet 351047c725f8488abbd074bd40790b48
1029 08:52:19.371075 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.377853 (+ 6778us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.411012 (+ 33159us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.419008 (+ 7996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1603518}
I20251029 08:52:19.419450 1176 tablet_replica.cc:333] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.419657 1176 raft_consensus.cc:2243] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.419684 1177 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.419812 1176 raft_consensus.cc:2272] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.419818 1177 ts_tablet_manager.cc:1403] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251029 08:52:19.420341 1176 ts_tablet_manager.cc:1916] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.420579 1177 raft_consensus.cc:359] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.421468 1177 raft_consensus.cc:385] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.421598 1177 raft_consensus.cc:740] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.423105 2016 raft_consensus.cc:493] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.423219 2016 raft_consensus.cc:515] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.423652 1974 tablet_replica.cc:333] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.423745 2016 leader_election.cc:290] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.423828 1974 raft_consensus.cc:2243] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.423964 1974 raft_consensus.cc:2272] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.425112 1974 ts_tablet_manager.cc:1916] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:19.424883 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4200db822deb4ace936f561c230ac1fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4200db822deb4ace936f561c230ac1fb already in progress: creating tablet
W20251029 08:52:19.425556 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4e3a2fab467348099a4a1fec74eb5e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3a2fab467348099a4a1fec74eb5e65 already in progress: creating tablet
W20251029 08:52:19.425931 861 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:19.418522 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b055a666e8cc467c861649776a8021e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:19.419590 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 351047c725f8488abbd074bd40790b48 (table default.Table_04d72f6fb117444a96f5d646217085e2 [id=de08f5f1950e4352a63c7d056bf4ce86]) successfully deleted
W20251029 08:52:19.430889 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b055a666e8cc467c861649776a8021e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:19.424285 1177 consensus_queue.cc:260] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.442162 2148 tablet.cc:1722] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:19.443534 1177 ts_tablet_manager.cc:1434] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251029 08:52:19.373265 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 491fb831cb2d4dbe9908fb6a571a43ca (DEFAULT_TABLE table=default.Table_22ad2d235d9c4ccdb70814d03790c887 [id=1daa44a8ad23428e9feb291be21d1eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.455914 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491fb831cb2d4dbe9908fb6a571a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.374533 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.380481 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52c35a09fd4b69b160ecaa2365ddd5 (DEFAULT_TABLE table=default.Table_d55bedf72b97413a8b66d1ab9a7ee591 [id=5ef1cde4a7104193982748c4e181f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.457069 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52c35a09fd4b69b160ecaa2365ddd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.353858 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 6f525da39ffa43c19838a0c039e4f96b (DEFAULT_TABLE table=default.Table_693b1227de8f476d97871a176deda38f [id=0ee770c0cc204c0aa7631288bca8aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.462839 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f525da39ffa43c19838a0c039e4f96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.463102 1974 ts_tablet_manager.cc:1929] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.463209 1974 log.cc:1199] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:19.463496 1974 ts_tablet_manager.cc:1950] T 363fcf8a3e0f4770a9743fddffa6495f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.466748 1176 ts_tablet_manager.cc:1929] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.385814 1179 ts_tablet_manager.cc:1929] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.466848 1176 log.cc:1199] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:19.466884 1179 log.cc:1199] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bc96ef1a94844a32923087c44a5bcfc8
I20251029 08:52:19.467123 1176 ts_tablet_manager.cc:1950] T 9e1c023b679b4c3298bf953b53d4eb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.467183 1179 ts_tablet_manager.cc:1950] T bc96ef1a94844a32923087c44a5bcfc8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.467845 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 746) took 1701 ms. Trace:
I20251029 08:52:19.467883 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 726) took 1383 ms. Trace:
I20251029 08:52:19.467957 1179 rpcz_store.cc:276] 1029 08:52:18.084482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.084548 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:19.319114 (+1234566us) ts_tablet_manager.cc:1151] Deleting tablet bc96ef1a94844a32923087c44a5bcfc8
1029 08:52:19.319126 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.381171 (+ 62045us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.385715 (+ 4544us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.467864 (+ 82149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234242}
I20251029 08:52:19.468274 1179 tablet_replica.cc:333] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.468431 1179 raft_consensus.cc:2243] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.468547 1179 raft_consensus.cc:2272] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.469013 1179 ts_tablet_manager.cc:1916] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.469249 1183 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.471045 1183 tablet_bootstrap.cc:654] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.388154 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 47f58be6d7f5492395ca6192f7ce0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:19.471961 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cf7604670833495c9dd61258a3301bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7604670833495c9dd61258a3301bb4 already in progress: creating tablet
I20251029 08:52:19.425086 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc9c221eaa9a4721ba25761206a13645" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.472214 1071 raft_consensus.cc:2468] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:19.472316 1974 tablet_replica.cc:333] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:19.472293 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 084a659932ec4996a6d6b160d67efdd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 084a659932ec4996a6d6b160d67efdd0 already in progress: creating tablet
I20251029 08:52:19.472492 1974 raft_consensus.cc:2243] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:19.472523 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 391977048e5f4f0b8f62e4d5e3d2946b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 391977048e5f4f0b8f62e4d5e3d2946b already in progress: creating tablet
I20251029 08:52:19.472663 1974 raft_consensus.cc:2272] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:19.472733 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2f36641b7086430a89380056e04b8aaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f36641b7086430a89380056e04b8aaf already in progress: creating tablet
I20251029 08:52:19.472693 861 leader_election.cc:304] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:19.472930 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 65eaf12e45c54bd1b0aedc647dc90436 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65eaf12e45c54bd1b0aedc647dc90436 already in progress: creating tablet
I20251029 08:52:19.473089 2046 raft_consensus.cc:2804] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 08:52:19.473106 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:19.473150 2046 raft_consensus.cc:493] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:19.473201 2046 raft_consensus.cc:3060] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.427829 2133 consensus_queue.cc:1048] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:19.473371 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:19.440461 2144 tablet.cc:1722] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:19.473577 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9a603e0ef22e48f18263ece15b799c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a603e0ef22e48f18263ece15b799c0c already in progress: creating tablet
I20251029 08:52:19.473775 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 363fcf8a3e0f4770a9743fddffa6495f (table default.Table_64f9de5287d54b348a406a42a7c379ab [id=0e01efd35bcd43dc88e2d92f0734b958]) successfully deleted
I20251029 08:52:19.474184 1177 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.476327 2046 raft_consensus.cc:515] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.478063 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc9c221eaa9a4721ba25761206a13645" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:19.478248 1071 raft_consensus.cc:3060] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.480262 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bc96ef1a94844a32923087c44a5bcfc8 (table default.Table_91e03c5d65a24eda8ba54f91a6be6081 [id=3299b4d85b5a4e76b0e927845299375d]) successfully deleted
I20251029 08:52:19.481082 1071 raft_consensus.cc:2468] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:19.481555 861 leader_election.cc:304] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:19.481854 2016 raft_consensus.cc:2804] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:19.481967 2016 raft_consensus.cc:697] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:19.482218 2016 consensus_queue.cc:237] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.483770 2046 leader_election.cc:290] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:19.484911 861 leader_election.cc:343] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:19.492158 2046 raft_consensus.cc:493] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.495002 2046 raft_consensus.cc:515] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.496608 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c3e50018f414f4b88e0e5d90cb7c925" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.496965 1071 raft_consensus.cc:2468] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:19.497265 2046 leader_election.cc:290] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.497718 861 leader_election.cc:304] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:19.498296 2156 raft_consensus.cc:2804] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:19.498397 2156 raft_consensus.cc:493] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:19.498459 2156 raft_consensus.cc:3060] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.467926 1176 rpcz_store.cc:276] 1029 08:52:17.766785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.772088 (+ 5303us) service_pool.cc:225] Handling call
1029 08:52:19.419401 (+1647313us) ts_tablet_manager.cc:1151] Deleting tablet 9e1c023b679b4c3298bf953b53d4eb96
1029 08:52:19.419410 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.451523 (+ 32113us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.466664 (+ 15141us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.467822 (+ 1158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647111}
I20251029 08:52:19.501467 1179 ts_tablet_manager.cc:1929] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.501672 1179 log.cc:1199] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/abc307b14aff4c5e937a5151aab1dfb9
I20251029 08:52:19.501741 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 750) took 1711 ms. Trace:
I20251029 08:52:19.501727 2156 raft_consensus.cc:515] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.502080 1179 ts_tablet_manager.cc:1950] T abc307b14aff4c5e937a5151aab1dfb9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.502707 2156 leader_election.cc:290] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.502681 1176 rpcz_store.cc:276] 1029 08:52:17.790374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.790424 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:19.501561 (+1711137us) ts_tablet_manager.cc:1151] Deleting tablet 0bd490da0b0e47fc965343c123a27b22
1029 08:52:19.501577 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.501719 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1710943}
I20251029 08:52:19.503398 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9e1c023b679b4c3298bf953b53d4eb96 (table default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]) successfully deleted
W20251029 08:52:19.504413 861 leader_election.cc:343] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7c3e50018f414f4b88e0e5d90cb7c925
W20251029 08:52:19.504518 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0bd490da0b0e47fc965343c123a27b22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bd490da0b0e47fc965343c123a27b22 already in progress: creating tablet
I20251029 08:52:19.504887 1176 tablet_replica.cc:333] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.505074 1176 raft_consensus.cc:2243] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.505117 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 727) took 1402 ms. Trace:
I20251029 08:52:19.505228 1176 raft_consensus.cc:2272] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.505265 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c3e50018f414f4b88e0e5d90cb7c925" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:19.505429 1070 raft_consensus.cc:3060] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.505767 1176 ts_tablet_manager.cc:1916] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:19.505861 861 leader_election.cc:343] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:19.505198 1179 rpcz_store.cc:276] 1029 08:52:18.102310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.102367 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:19.468235 (+1365868us) ts_tablet_manager.cc:1151] Deleting tablet abc307b14aff4c5e937a5151aab1dfb9
1029 08:52:19.468243 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.471975 (+ 3732us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.501357 (+ 29382us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.505097 (+ 3740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365673}
I20251029 08:52:19.508090 1070 raft_consensus.cc:2468] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:19.508221 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet abc307b14aff4c5e937a5151aab1dfb9 (table default.Table_6d2b9c809be24c519e176adfd43dfa45 [id=6058973c45824d5ab4b2071bc573ae1d]) successfully deleted
I20251029 08:52:19.508503 1179 tablet_replica.cc:333] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.508605 861 leader_election.cc:304] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:19.508703 1179 raft_consensus.cc:2243] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.473201 1974 ts_tablet_manager.cc:1916] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.508831 1179 raft_consensus.cc:2272] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.509301 1179 ts_tablet_manager.cc:1916] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.509406 2156 raft_consensus.cc:2804] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:19.509552 2156 raft_consensus.cc:697] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:19.509889 2156 consensus_queue.cc:237] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.517529 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52c35a09fd4b69b160ecaa2365ddd5 (DEFAULT_TABLE table=default.Table_d55bedf72b97413a8b66d1ab9a7ee591 [id=5ef1cde4a7104193982748c4e181f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.517971 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52c35a09fd4b69b160ecaa2365ddd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.527200 1179 ts_tablet_manager.cc:1929] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.528101 1177 tablet_bootstrap.cc:654] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.530025 1050 tablet_service.cc:1505] Processing CreateTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 (DEFAULT_TABLE table=default.Table_14f731cef6024ab79260c04bef01d3b5 [id=cff5e0a9f50d48f98a400a293eb0163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.534189 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbad8c4014b6423cb20e3e5c22ed69f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.535099 2038 raft_consensus.cc:493] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.535236 2038 raft_consensus.cc:515] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.535953 2038 leader_election.cc:290] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.536952 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af38662420d14ac1a7cb5fd03fc1b9d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.537572 720 leader_election.cc:343] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.540592 1179 log.cc:1199] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/215cd666c71a4179a19e298868a88e96
I20251029 08:52:19.540980 1179 ts_tablet_manager.cc:1950] T 215cd666c71a4179a19e298868a88e96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.541126 748 tablet_service.cc:1505] Processing CreateTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 (DEFAULT_TABLE table=default.Table_14f731cef6024ab79260c04bef01d3b5 [id=cff5e0a9f50d48f98a400a293eb0163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.541589 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbad8c4014b6423cb20e3e5c22ed69f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.541702 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 728) took 1436 ms. Trace:
I20251029 08:52:19.541792 1179 rpcz_store.cc:276] 1029 08:52:18.105145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.105193 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:19.508458 (+1403265us) ts_tablet_manager.cc:1151] Deleting tablet 215cd666c71a4179a19e298868a88e96
1029 08:52:19.508466 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.523866 (+ 15400us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.527119 (+ 3253us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.541685 (+ 14566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403101}
I20251029 08:52:19.542071 1179 tablet_replica.cc:333] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.542224 1179 raft_consensus.cc:2243] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.542338 1179 raft_consensus.cc:2272] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.542301 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 215cd666c71a4179a19e298868a88e96 (table default.Table_a58b4ac66df64b0db81210ce141ece47 [id=88bde60591764b37b719bdeea15ef0cd]) successfully deleted
I20251029 08:52:19.542783 1179 ts_tablet_manager.cc:1916] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.546006 1176 ts_tablet_manager.cc:1929] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.536840 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af38662420d14ac1a7cb5fd03fc1b9d8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:19.544238 1974 ts_tablet_manager.cc:1929] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.553042 1050 tablet_service.cc:1505] Processing CreateTablet for tablet d9cabbbac8254a949750244e1d6991eb (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.553478 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cabbbac8254a949750244e1d6991eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.553045 1974 log.cc:1199] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:19.554153 1974 ts_tablet_manager.cc:1950] T d30183814f5e4f2ab9d71fe493eb801c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.559635 887 tablet_service.cc:1505] Processing CreateTablet for tablet d9cabbbac8254a949750244e1d6991eb (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.560312 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cabbbac8254a949750244e1d6991eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.567512 1179 ts_tablet_manager.cc:1929] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.569010 893 tablet_service.cc:1505] Processing CreateTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 (DEFAULT_TABLE table=default.Table_14f731cef6024ab79260c04bef01d3b5 [id=cff5e0a9f50d48f98a400a293eb0163c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.569272 1179 log.cc:1199] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ee8117e04f5f40e98ea755dc461067dd
I20251029 08:52:19.569411 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbad8c4014b6423cb20e3e5c22ed69f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.569669 1179 ts_tablet_manager.cc:1950] T ee8117e04f5f40e98ea755dc461067dd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.570406 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 729) took 1464 ms. Trace:
I20251029 08:52:19.570555 1179 rpcz_store.cc:276] 1029 08:52:18.105534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.105580 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:19.542033 (+1436453us) ts_tablet_manager.cc:1151] Deleting tablet ee8117e04f5f40e98ea755dc461067dd
1029 08:52:19.542041 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.545794 (+ 3753us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.567406 (+ 21612us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.570388 (+ 2982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436327}
I20251029 08:52:19.576515 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ee8117e04f5f40e98ea755dc461067dd (table default.Table_d6dc6ba680874a51afe94c07adb83e2b [id=5730455f82a94d8698ac62979d87d9d6]) successfully deleted
I20251029 08:52:19.576990 1179 tablet_replica.cc:333] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.577232 1179 raft_consensus.cc:2243] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.577462 1179 raft_consensus.cc:2272] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.578071 1179 ts_tablet_manager.cc:1916] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.571413 1183 tablet_bootstrap.cc:492] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.578573 1183 ts_tablet_manager.cc:1403] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.002s
I20251029 08:52:19.579383 1183 raft_consensus.cc:359] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.579674 1183 raft_consensus.cc:385] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.579768 1183 raft_consensus.cc:740] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.580060 1183 consensus_queue.cc:260] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.584956 1183 ts_tablet_manager.cc:1434] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251029 08:52:19.590790 1179 ts_tablet_manager.cc:1929] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.591064 1179 log.cc:1199] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/5a7a673ba4fd4abcbfc3493c38dcc257
I20251029 08:52:19.591435 1179 ts_tablet_manager.cc:1950] T 5a7a673ba4fd4abcbfc3493c38dcc257 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.591934 1183 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.593396 1183 tablet_bootstrap.cc:654] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.596477 1050 tablet_service.cc:1505] Processing CreateTablet for tablet bda184ab18764ae592a6a3a25ef9a724 (DEFAULT_TABLE table=default.Table_713096b122044a3884d7d40bf3d84cb0 [id=1d3cb945e29741e5921c9f716add675b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.598109 1036 tablet_service.cc:1505] Processing CreateTablet for tablet b7c6b22df08e4aed93dfbcd7aa3cc280 (DEFAULT_TABLE table=default.Table_0956bbcc718e4744ab316e8efd03b1fa [id=cd0bed7efb264db798d541cd0e92ce48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.599324 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 730) took 1493 ms. Trace:
I20251029 08:52:19.600371 1179 rpcz_store.cc:276] 1029 08:52:18.105853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.105890 (+ 37us) service_pool.cc:225] Handling call
1029 08:52:19.576935 (+1471045us) ts_tablet_manager.cc:1151] Deleting tablet 5a7a673ba4fd4abcbfc3493c38dcc257
1029 08:52:19.576945 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.587662 (+ 10717us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.590676 (+ 3014us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.599296 (+ 8620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470903}
I20251029 08:52:19.601567 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 5a7a673ba4fd4abcbfc3493c38dcc257 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:19.602381 1974 tablet_replica.cc:333] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.603369 1974 raft_consensus.cc:2243] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.603850 1974 raft_consensus.cc:2272] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.604877 1974 ts_tablet_manager.cc:1916] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.611482 1974 ts_tablet_manager.cc:1929] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.614836 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda184ab18764ae592a6a3a25ef9a724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.615746 1974 log.cc:1199] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/072cf27b0743488c8166e03b5e61c545
I20251029 08:52:19.616125 1974 ts_tablet_manager.cc:1950] T 072cf27b0743488c8166e03b5e61c545 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.616739 1974 tablet_replica.cc:333] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.616904 1974 raft_consensus.cc:2243] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.617062 1974 raft_consensus.cc:2272] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.617594 1974 ts_tablet_manager.cc:1916] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.617681 1179 tablet_replica.cc:333] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.617854 1179 raft_consensus.cc:2243] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.618005 1179 raft_consensus.cc:2272] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.618503 1179 ts_tablet_manager.cc:1916] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.623836 1974 ts_tablet_manager.cc:1929] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.624045 1974 log.cc:1199] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/5385fef404a4463595074f0c8ff822d7
I20251029 08:52:19.624197 2016 raft_consensus.cc:493] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.624449 1974 ts_tablet_manager.cc:1950] T 5385fef404a4463595074f0c8ff822d7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.624346 2016 raft_consensus.cc:515] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.625077 2016 leader_election.cc:290] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.625845 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8991ea78321545be85d5c480841e4751" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.626335 861 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 8991ea78321545be85d5c480841e4751
W20251029 08:52:19.626544 861 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.626626 861 leader_election.cc:304] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:19.627038 2016 raft_consensus.cc:2749] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.627801 1974 tablet_replica.cc:333] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.628080 1974 raft_consensus.cc:2243] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.628372 1974 pending_rounds.cc:70] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:19.628470 1974 pending_rounds.cc:77] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037637370793984 op_type: NO_OP noop_request { }
I20251029 08:52:19.628645 1974 raft_consensus.cc:2889] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.628779 1974 raft_consensus.cc:2272] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.629537 1974 ts_tablet_manager.cc:1916] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.629721 1179 ts_tablet_manager.cc:1929] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.629875 1179 log.cc:1199] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a8c5008d9df9419bb4b6d678c6dc27a6
I20251029 08:52:19.630228 1179 ts_tablet_manager.cc:1950] T a8c5008d9df9419bb4b6d678c6dc27a6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.631000 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 731) took 1519 ms. Trace:
I20251029 08:52:19.631138 1179 rpcz_store.cc:276] 1029 08:52:18.112001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.112074 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:19.602227 (+1490153us) ts_tablet_manager.cc:1151] Deleting tablet a8c5008d9df9419bb4b6d678c6dc27a6
1029 08:52:19.617619 (+ 15392us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.622514 (+ 4895us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.629624 (+ 7110us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.630979 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489958}
I20251029 08:52:19.631750 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a8c5008d9df9419bb4b6d678c6dc27a6 (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:19.632272 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 732) took 1518 ms. Trace:
I20251029 08:52:19.632402 1179 rpcz_store.cc:276] 1029 08:52:18.113762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.113808 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:19.632139 (+1518331us) ts_tablet_manager.cc:1151] Deleting tablet 4200db822deb4ace936f561c230ac1fb
1029 08:52:19.632150 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.632254 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518182}
W20251029 08:52:19.632938 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4200db822deb4ace936f561c230ac1fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4200db822deb4ace936f561c230ac1fb already in progress: creating tablet
I20251029 08:52:19.633446 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 733) took 1519 ms. Trace:
I20251029 08:52:19.635051 893 tablet_service.cc:1505] Processing CreateTablet for tablet bda184ab18764ae592a6a3a25ef9a724 (DEFAULT_TABLE table=default.Table_713096b122044a3884d7d40bf3d84cb0 [id=1d3cb945e29741e5921c9f716add675b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.635442 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda184ab18764ae592a6a3a25ef9a724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.642597 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d30183814f5e4f2ab9d71fe493eb801c (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
I20251029 08:52:19.642987 887 tablet_service.cc:1505] Processing CreateTablet for tablet b7c6b22df08e4aed93dfbcd7aa3cc280 (DEFAULT_TABLE table=default.Table_0956bbcc718e4744ab316e8efd03b1fa [id=cd0bed7efb264db798d541cd0e92ce48]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:19.643136 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 85df43bf781c421688f163331de0eb2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85df43bf781c421688f163331de0eb2b already in progress: creating tablet
I20251029 08:52:19.643419 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6b22df08e4aed93dfbcd7aa3cc280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.643692 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 072cf27b0743488c8166e03b5e61c545 (table default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]) successfully deleted
I20251029 08:52:19.643959 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 5385fef404a4463595074f0c8ff822d7 (table default.Table_58fa7040e0e2467588dd8b5a0a2eba4b [id=2cfb2b85354d43e7a203d35a4448346d]) successfully deleted
I20251029 08:52:19.644503 1179 rpcz_store.cc:276] 1029 08:52:18.114148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.114192 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:19.633293 (+1519101us) ts_tablet_manager.cc:1151] Deleting tablet 4e3a2fab467348099a4a1fec74eb5e65
1029 08:52:19.633303 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.633428 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518966}
W20251029 08:52:19.645076 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4e3a2fab467348099a4a1fec74eb5e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3a2fab467348099a4a1fec74eb5e65 already in progress: creating tablet
I20251029 08:52:19.645537 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 734) took 1528 ms. Trace:
I20251029 08:52:19.645707 1179 rpcz_store.cc:276] 1029 08:52:18.117131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.117174 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:19.645385 (+1528211us) ts_tablet_manager.cc:1151] Deleting tablet cf7604670833495c9dd61258a3301bb4
1029 08:52:19.645398 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.645515 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528061}
W20251029 08:52:19.646267 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cf7604670833495c9dd61258a3301bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7604670833495c9dd61258a3301bb4 already in progress: creating tablet
I20251029 08:52:19.650086 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 735) took 1160 ms. Trace:
I20251029 08:52:19.650244 1179 rpcz_store.cc:276] 1029 08:52:18.489647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.489701 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:19.649945 (+1160244us) ts_tablet_manager.cc:1151] Deleting tablet 084a659932ec4996a6d6b160d67efdd0
1029 08:52:19.649958 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.650063 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160070}
W20251029 08:52:19.650933 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 084a659932ec4996a6d6b160d67efdd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 084a659932ec4996a6d6b160d67efdd0 already in progress: creating tablet
I20251029 08:52:19.651441 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 736) took 1161 ms. Trace:
I20251029 08:52:19.651556 1179 rpcz_store.cc:276] 1029 08:52:18.490012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.490056 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:19.651244 (+1161188us) ts_tablet_manager.cc:1151] Deleting tablet 391977048e5f4f0b8f62e4d5e3d2946b
1029 08:52:19.651255 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.651423 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161050}
I20251029 08:52:19.651939 1974 ts_tablet_manager.cc:1929] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.652117 1974 log.cc:1199] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/07906300f1d54d4e8fde5165ee14c44f
W20251029 08:52:19.652096 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 391977048e5f4f0b8f62e4d5e3d2946b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 391977048e5f4f0b8f62e4d5e3d2946b already in progress: creating tablet
I20251029 08:52:19.652489 1974 ts_tablet_manager.cc:1950] T 07906300f1d54d4e8fde5165ee14c44f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.652524 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 737) took 1162 ms. Trace:
I20251029 08:52:19.652627 1179 rpcz_store.cc:276] 1029 08:52:18.490303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.490345 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:19.652413 (+1162068us) ts_tablet_manager.cc:1151] Deleting tablet 61e96f14b6c74450828ef50148ab7ac5
1029 08:52:19.652424 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.652509 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161925}
I20251029 08:52:19.653077 1974 tablet_replica.cc:333] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:19.653105 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:19.653296 1974 raft_consensus.cc:2243] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.653483 1974 raft_consensus.cc:2272] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.634307 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6b22df08e4aed93dfbcd7aa3cc280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.654029 1974 ts_tablet_manager.cc:1916] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:19.553365 718 leader_election.cc:343] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.656133 718 leader_election.cc:304] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:19.656510 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 103158us
I20251029 08:52:19.568432 1178 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.656985 2038 raft_consensus.cc:2749] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.657735 1178 ts_tablet_manager.cc:1403] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.255s user 0.014s sys 0.013s
I20251029 08:52:19.658185 1070 raft_consensus.cc:3060] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.658577 1178 raft_consensus.cc:359] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:19.658648 583 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b -> Peer f718bd0f00004f20a89d028a2fa8e6b9 (127.0.104.62:40435): Couldn't send request to peer f718bd0f00004f20a89d028a2fa8e6b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:40435: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251029 08:52:19.658748 1178 raft_consensus.cc:385] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.658792 1178 raft_consensus.cc:740] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.658979 1178 consensus_queue.cc:260] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.659471 1178 ts_tablet_manager.cc:1434] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:19.659750 1178 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.664197 1178 tablet_bootstrap.cc:654] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.666862 1177 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.667011 1177 ts_tablet_manager.cc:1403] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.223s user 0.004s sys 0.001s
I20251029 08:52:19.667749 1177 raft_consensus.cc:359] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.667917 1177 raft_consensus.cc:385] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.667972 1177 raft_consensus.cc:740] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.665097 757 tablet_service.cc:1505] Processing CreateTablet for tablet d9cabbbac8254a949750244e1d6991eb (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.665530 745 tablet_service.cc:1505] Processing CreateTablet for tablet b7c6b22df08e4aed93dfbcd7aa3cc280 (DEFAULT_TABLE table=default.Table_0956bbcc718e4744ab316e8efd03b1fa [id=cd0bed7efb264db798d541cd0e92ce48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.665540 758 tablet_service.cc:1505] Processing CreateTablet for tablet bda184ab18764ae592a6a3a25ef9a724 (DEFAULT_TABLE table=default.Table_713096b122044a3884d7d40bf3d84cb0 [id=1d3cb945e29741e5921c9f716add675b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.668499 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cabbbac8254a949750244e1d6991eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.668768 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6b22df08e4aed93dfbcd7aa3cc280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.672574 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda184ab18764ae592a6a3a25ef9a724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.673730 2044 consensus_queue.cc:1048] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:19.678738 1176 log.cc:1199] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:19.679502 1176 ts_tablet_manager.cc:1950] T bbc4345cabbc476485b6a040a076ddaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.689045 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 751) took 1842 ms. Trace:
I20251029 08:52:19.696295 1176 rpcz_store.cc:276] 1029 08:52:17.846197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.846654 (+ 457us) service_pool.cc:225] Handling call
1029 08:52:19.504836 (+1658182us) ts_tablet_manager.cc:1151] Deleting tablet bbc4345cabbc476485b6a040a076ddaf
1029 08:52:19.504848 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.532848 (+ 28000us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.545925 (+ 13077us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.689015 (+143090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1657862}
I20251029 08:52:19.697412 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bbc4345cabbc476485b6a040a076ddaf (table default.Table_64f9de5287d54b348a406a42a7c379ab [id=0e01efd35bcd43dc88e2d92f0734b958]) successfully deleted
I20251029 08:52:19.697881 1176 tablet_replica.cc:333] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.698347 1176 raft_consensus.cc:2243] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.699111 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:19.700855 1176 raft_consensus.cc:2272] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.701413 1176 ts_tablet_manager.cc:1916] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.703161 593 master_service.cc:954] ReplaceTablet: received request to replace tablet a013257089774ba8b1f535e5f94ac1bb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.704581 588 master_service.cc:954] ReplaceTablet: received request to replace tablet ff1b7181e52041288c45b053124bd844 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.705155 1178 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.705281 1178 ts_tablet_manager.cc:1403] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.004s
I20251029 08:52:19.705933 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bd00dc7aa1464015aafad884320e9740" } new_table_name: "default.Table_f87314ab36174c9184d9a2cb9e00b51f" modify_external_catalogs: true
I20251029 08:52:19.706111 1178 raft_consensus.cc:359] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.706280 1178 raft_consensus.cc:385] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.706329 1178 raft_consensus.cc:740] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.706559 1178 consensus_queue.cc:260] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.707285 1178 ts_tablet_manager.cc:1434] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:19.707551 1178 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.709177 1178 tablet_bootstrap.cc:654] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.718575 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 363fcf8a3e0f4770a9743fddffa6495f with delete_type TABLET_DATA_DELETED (replaced by tablet bbc4345cabbc476485b6a040a076ddaf) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.718736 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 608b7095d42941ccb0c4eb57f6754e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.719383 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:19.721908 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_84f0a37a39f64def888bada4afc4ccb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:19.723022 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84f0a37a39f64def888bada4afc4ccb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:19.725059 582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 51591.6us
I20251029 08:52:19.729066 2144 tablet.cc:1722] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:19.739660 2044 raft_consensus.cc:493] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.739814 2044 raft_consensus.cc:515] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.740461 2044 leader_election.cc:290] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:19.741643 996 leader_election.cc:343] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 608b7095d42941ccb0c4eb57f6754e74
W20251029 08:52:19.742254 998 leader_election.cc:343] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:19.742355 998 leader_election.cc:304] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.742692 2044 raft_consensus.cc:2749] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.743716 1178 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.743862 1178 ts_tablet_manager.cc:1403] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251029 08:52:19.746017 1178 raft_consensus.cc:359] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.746222 1178 raft_consensus.cc:385] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.748076 1178 raft_consensus.cc:740] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.748373 1178 consensus_queue.cc:260] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.749282 1178 ts_tablet_manager.cc:1434] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:19.749732 1178 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.750275 2016 raft_consensus.cc:493] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.750449 2016 raft_consensus.cc:515] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.751081 2016 leader_election.cc:290] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.751446 1178 tablet_bootstrap.cc:654] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:19.756124 861 leader_election.cc:343] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:19.756832 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 07906300f1d54d4e8fde5165ee14c44f (table default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]) successfully deleted
I20251029 08:52:19.757529 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.757797 887 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9c221eaa9a4721ba25761206a13645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.758231 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.762176 1974 ts_tablet_manager.cc:1929] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.762351 1974 log.cc:1199] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:19.762746 1974 ts_tablet_manager.cc:1950] T c572c68b32d443b2a019e1f4bf2d82fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.763542 1974 tablet_replica.cc:333] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.763819 1974 raft_consensus.cc:2243] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.764169 1974 pending_rounds.cc:70] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:19.764235 1974 pending_rounds.cc:77] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037640234618880 op_type: NO_OP noop_request { }
I20251029 08:52:19.765720 1178 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.765954 1178 ts_tablet_manager.cc:1403] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20251029 08:52:19.767093 1178 raft_consensus.cc:359] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.767407 1178 raft_consensus.cc:385] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.767522 1178 raft_consensus.cc:740] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.767854 1178 consensus_queue.cc:260] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.769448 1177 consensus_queue.cc:260] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.769549 1176 ts_tablet_manager.cc:1929] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.769696 1176 log.cc:1199] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:19.769995 1176 ts_tablet_manager.cc:1950] T c66cbb495f5142d79a674f7d5adc505e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.770720 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 752) took 1868 ms. Trace:
I20251029 08:52:19.770797 1176 rpcz_store.cc:276] 1029 08:52:17.902751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:17.902803 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:19.697819 (+1795016us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:19.697832 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.738479 (+ 40647us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.769407 (+ 30928us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.770699 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794857}
I20251029 08:52:19.771118 1176 tablet_replica.cc:333] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.771304 1176 raft_consensus.cc:2243] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.771431 1176 raft_consensus.cc:2272] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.771916 1176 ts_tablet_manager.cc:1916] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.774024 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c66cbb495f5142d79a674f7d5adc505e (table default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]) successfully deleted
I20251029 08:52:19.775842 1178 ts_tablet_manager.cc:1434] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251029 08:52:19.776265 1178 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.778751 1178 tablet_bootstrap.cc:654] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.764369 1974 raft_consensus.cc:2889] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.779347 1974 raft_consensus.cc:2272] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.780717 1974 ts_tablet_manager.cc:1916] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.785897 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.786477 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.788383 1177 ts_tablet_manager.cc:1434] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.121s user 0.002s sys 0.000s
I20251029 08:52:19.793475 1974 ts_tablet_manager.cc:1929] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.795308 1178 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.795620 1178 ts_tablet_manager.cc:1403] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251029 08:52:19.796588 1178 raft_consensus.cc:359] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.796864 1178 raft_consensus.cc:385] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.796964 1178 raft_consensus.cc:740] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.797344 1178 consensus_queue.cc:260] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.800902 1178 ts_tablet_manager.cc:1434] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:19.802104 1178 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.803088 1176 ts_tablet_manager.cc:1929] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.804116 1974 log.cc:1199] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:19.804594 1974 ts_tablet_manager.cc:1950] T 7c3e50018f414f4b88e0e5d90cb7c925 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.805373 1176 log.cc:1199] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:19.805426 1974 tablet_replica.cc:333] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.805723 1176 ts_tablet_manager.cc:1950] T eed66c3fa3f848df905a2a31fd553474 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.805737 1974 raft_consensus.cc:2243] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.806190 1974 raft_consensus.cc:2272] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.806537 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 754) took 1789 ms. Trace:
I20251029 08:52:19.806648 1176 rpcz_store.cc:276] 1029 08:52:18.017425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.017501 (+ 76us) service_pool.cc:225] Handling call
1029 08:52:19.771070 (+1753569us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:19.771081 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.794877 (+ 23796us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.802964 (+ 8087us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.806515 (+ 3551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637397}
I20251029 08:52:19.807021 1974 ts_tablet_manager.cc:1916] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.807086 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 755) took 1729 ms. Trace:
I20251029 08:52:19.807173 1176 rpcz_store.cc:276] 1029 08:52:18.077269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.077346 (+ 77us) service_pool.cc:225] Handling call
1029 08:52:19.806955 (+1729609us) ts_tablet_manager.cc:1151] Deleting tablet 4200db822deb4ace936f561c230ac1fb
1029 08:52:19.806965 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.807070 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673058}
I20251029 08:52:19.807489 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 756) took 1729 ms. Trace:
I20251029 08:52:19.807565 1176 rpcz_store.cc:276] 1029 08:52:18.077691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.077745 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:19.807389 (+1729644us) ts_tablet_manager.cc:1151] Deleting tablet 4e3a2fab467348099a4a1fec74eb5e65
1029 08:52:19.807397 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.807475 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673438}
I20251029 08:52:19.807859 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 757) took 1729 ms. Trace:
I20251029 08:52:19.807936 1176 rpcz_store.cc:276] 1029 08:52:18.077982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.078031 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:19.807767 (+1729736us) ts_tablet_manager.cc:1151] Deleting tablet cf7604670833495c9dd61258a3301bb4
1029 08:52:19.807775 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.807844 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673775}
I20251029 08:52:19.808243 1176 tablet_replica.cc:333] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.808495 1176 raft_consensus.cc:2243] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.808508 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet eed66c3fa3f848df905a2a31fd553474 (table default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]) successfully deleted
I20251029 08:52:19.808696 1176 raft_consensus.cc:2272] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:19.808848 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4200db822deb4ace936f561c230ac1fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4200db822deb4ace936f561c230ac1fb already in progress: creating tablet
W20251029 08:52:19.809031 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4e3a2fab467348099a4a1fec74eb5e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3a2fab467348099a4a1fec74eb5e65 already in progress: creating tablet
W20251029 08:52:19.809226 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cf7604670833495c9dd61258a3301bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7604670833495c9dd61258a3301bb4 already in progress: creating tablet
I20251029 08:52:19.809374 1176 ts_tablet_manager.cc:1916] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.803927 1178 tablet_bootstrap.cc:654] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.814198 1974 ts_tablet_manager.cc:1929] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:19.814391 1974 log.cc:1199] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c4e1b6b6359b437aa295b473fa42dce1
I20251029 08:52:19.814802 1974 ts_tablet_manager.cc:1950] T c4e1b6b6359b437aa295b473fa42dce1 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.816207 1974 tablet_replica.cc:333] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.816480 1974 raft_consensus.cc:2243] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.818249 1974 raft_consensus.cc:2272] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.818884 1974 ts_tablet_manager.cc:1916] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.820438 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 738) took 1329 ms. Trace:
I20251029 08:52:19.820577 1179 rpcz_store.cc:276] 1029 08:52:18.490625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.490664 (+ 39us) service_pool.cc:225] Handling call
1029 08:52:19.820241 (+1329577us) ts_tablet_manager.cc:1151] Deleting tablet 2f36641b7086430a89380056e04b8aaf
1029 08:52:19.820254 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.820414 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329411}
I20251029 08:52:19.821012 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 739) took 1330 ms. Trace:
W20251029 08:52:19.821151 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2f36641b7086430a89380056e04b8aaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f36641b7086430a89380056e04b8aaf already in progress: creating tablet
I20251029 08:52:19.821157 1179 rpcz_store.cc:276] 1029 08:52:18.490920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.490958 (+ 38us) service_pool.cc:225] Handling call
1029 08:52:19.820891 (+1329933us) ts_tablet_manager.cc:1151] Deleting tablet 65eaf12e45c54bd1b0aedc647dc90436
1029 08:52:19.820901 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.820994 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329633}
I20251029 08:52:19.821419 1179 tablet_replica.cc:333] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.821612 1179 raft_consensus.cc:2243] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.821785 1179 raft_consensus.cc:2272] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:19.821775 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 65eaf12e45c54bd1b0aedc647dc90436 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65eaf12e45c54bd1b0aedc647dc90436 already in progress: creating tablet
I20251029 08:52:19.822316 1179 ts_tablet_manager.cc:1916] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.824213 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "787d6d7f1870450fa21cb9cba34e72fe" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:19.824734 861 leader_election.cc:343] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.824858 861 leader_election.cc:304] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:19.825028 861 raft_consensus.cc:2686] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:19.825357 2165 raft_consensus.cc:493] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.825496 2165 raft_consensus.cc:515] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.826160 2165 leader_election.cc:290] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:19.826422 1177 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.827261 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5051ff776e4975b917db0393a1f945" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:19.827836 718 leader_election.cc:343] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.827950 1177 tablet_bootstrap.cc:654] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.828199 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5051ff776e4975b917db0393a1f945" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.829228 887 tablet_service.cc:1505] Processing CreateTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.829605 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.830178 1974 ts_tablet_manager.cc:1929] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.830264 1974 log.cc:1199] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:19.830543 1974 ts_tablet_manager.cc:1950] T 787d6d7f1870450fa21cb9cba34e72fe P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:19.831444 720 leader_election.cc:343] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.831563 720 leader_election.cc:304] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.832149 2165 raft_consensus.cc:2749] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.832621 1974 tablet_replica.cc:333] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.832864 1974 raft_consensus.cc:2243] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:19.833184 1974 pending_rounds.cc:70] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:19.833292 1974 pending_rounds.cc:77] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037640121421824 op_type: NO_OP noop_request { }
I20251029 08:52:19.833473 1974 raft_consensus.cc:2889] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:19.833619 1974 raft_consensus.cc:2272] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.834762 1974 ts_tablet_manager.cc:1916] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.838990 1176 ts_tablet_manager.cc:1929] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.839123 1176 log.cc:1199] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/fc04616301354cfe873c825a9f5d7ab6
W20251029 08:52:19.840652 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8052ae8345ad4f3f95714d7b9173d3d8 consensus state has no leader
I20251029 08:52:19.841364 1974 ts_tablet_manager.cc:1929] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:19.841467 1974 log.cc:1199] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:19.841812 1974 ts_tablet_manager.cc:1950] T bc9c221eaa9a4721ba25761206a13645 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.842437 1974 tablet_replica.cc:333] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.842686 1974 raft_consensus.cc:2243] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.842872 1974 raft_consensus.cc:2272] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.843304 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]). Expected version 2 got 0
I20251029 08:52:19.843562 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:19.843676 1974 ts_tablet_manager.cc:1916] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.846181 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d2fa4bcfb5a240239a27ceb1bebf974c" } modify_external_catalogs: true
I20251029 08:52:19.846401 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d2fa4bcfb5a240239a27ceb1bebf974c" } modify_external_catalogs: true
I20251029 08:52:19.848126 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7f4559dca42a45bd9d4f2e050a759faa" } modify_external_catalogs: true
I20251029 08:52:19.848297 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7f4559dca42a45bd9d4f2e050a759faa" } modify_external_catalogs: true
I20251029 08:52:19.851470 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_343f0a80fc4c4dceb7098e5f75a81389" } modify_external_catalogs: true
I20251029 08:52:19.851224 1974 ts_tablet_manager.cc:1929] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.851660 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_343f0a80fc4c4dceb7098e5f75a81389" } modify_external_catalogs: true
I20251029 08:52:19.851727 1974 log.cc:1199] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/8991ea78321545be85d5c480841e4751
I20251029 08:52:19.852092 1974 ts_tablet_manager.cc:1950] T 8991ea78321545be85d5c480841e4751 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.854811 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2184d67d2edc408f939c2b4114a7eb15" } modify_external_catalogs: true
I20251029 08:52:19.855018 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2184d67d2edc408f939c2b4114a7eb15" } modify_external_catalogs: true
I20251029 08:52:19.861100 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c572c68b32d443b2a019e1f4bf2d82fe (table default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]) successfully deleted
W20251029 08:52:19.861667 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bda3e143699b41b6bba308c83bbb3e0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bda3e143699b41b6bba308c83bbb3e0d already in progress: creating tablet
I20251029 08:52:19.862049 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7c3e50018f414f4b88e0e5d90cb7c925 (table default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]) successfully deleted
W20251029 08:52:19.862392 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cf09e8fd3d2b43318fa37d922c8b826f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf09e8fd3d2b43318fa37d922c8b826f already in progress: creating tablet
W20251029 08:52:19.862736 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 because the tablet was not found. No further retry: Not found: Tablet not found: 62b7fe26c7ad48cdb935e8dc6b1962a7
I20251029 08:52:19.863047 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
W20251029 08:52:19.863363 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e14d05e64689423da77d6d60b760cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d05e64689423da77d6d60b760cc06 already in progress: creating tablet
W20251029 08:52:19.863701 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 663d6587a5e14519afbb0f94d691d1b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663d6587a5e14519afbb0f94d691d1b4 already in progress: creating tablet
W20251029 08:52:19.864003 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 47f58be6d7f5492395ca6192f7ce0930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47f58be6d7f5492395ca6192f7ce0930 already in progress: creating tablet
I20251029 08:52:19.864360 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 787d6d7f1870450fa21cb9cba34e72fe (table default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]) successfully deleted
I20251029 08:52:19.864758 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bc9c221eaa9a4721ba25761206a13645 (table default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]) successfully deleted
I20251029 08:52:19.865054 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 8991ea78321545be85d5c480841e4751 (table default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]) successfully deleted
I20251029 08:52:19.865777 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.866716 1974 tablet_replica.cc:333] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.866987 1974 raft_consensus.cc:2243] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.867225 1974 raft_consensus.cc:2272] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.868207 1974 ts_tablet_manager.cc:1916] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.871520 1176 ts_tablet_manager.cc:1950] T fc04616301354cfe873c825a9f5d7ab6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.875890 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 758) took 1648 ms. Trace:
I20251029 08:52:19.876938 1176 rpcz_store.cc:276] 1029 08:52:18.227345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.227402 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:19.808190 (+1580788us) ts_tablet_manager.cc:1151] Deleting tablet fc04616301354cfe873c825a9f5d7ab6
1029 08:52:19.808201 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.825176 (+ 16975us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.838881 (+ 13705us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.875853 (+ 36972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1580581}
I20251029 08:52:19.877732 1176 tablet_replica.cc:333] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.877938 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet fc04616301354cfe873c825a9f5d7ab6 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
I20251029 08:52:19.878280 1176 raft_consensus.cc:2243] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.878754 1176 raft_consensus.cc:2272] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.879648 1176 ts_tablet_manager.cc:1916] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.888549 748 tablet_service.cc:1505] Processing CreateTablet for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 (DEFAULT_TABLE table=default.Table_1f5f2cd9e3f14a0a8f3316489f46929a [id=e74d6902ea624d2ea09dde8448cdea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:19.889046 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:19.889621 1974 ts_tablet_manager.cc:1929] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.890823 1974 log.cc:1199] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/840ff254244f43a18679232279e02d1f
I20251029 08:52:19.891299 1974 ts_tablet_manager.cc:1950] T 840ff254244f43a18679232279e02d1f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.892304 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 840ff254244f43a18679232279e02d1f (table default.Table_82ec1f06f21147c991f019803452df8c [id=383565dcefdd479288ddfc261cbced6f]) successfully deleted
I20251029 08:52:19.895081 1176 ts_tablet_manager.cc:1929] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.896317 1176 log.cc:1199] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:19.896952 1176 ts_tablet_manager.cc:1950] T 084a659932ec4996a6d6b160d67efdd0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.898370 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 759) took 1489 ms. Trace:
I20251029 08:52:19.899585 1176 rpcz_store.cc:276] 1029 08:52:18.408676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.408797 (+ 121us) service_pool.cc:225] Handling call
1029 08:52:19.877666 (+1468869us) ts_tablet_manager.cc:1151] Deleting tablet 084a659932ec4996a6d6b160d67efdd0
1029 08:52:19.877680 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.887818 (+ 10138us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.894970 (+ 7152us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.898340 (+ 3370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429398}
I20251029 08:52:19.900786 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 084a659932ec4996a6d6b160d67efdd0 (table default.Table_c36b90fc430a4a89ba87469be66cb2a5 [id=d88876b1c5514d78892db64b612182ba]) successfully deleted
I20251029 08:52:19.901405 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 760) took 1469 ms. Trace:
I20251029 08:52:19.901894 1176 rpcz_store.cc:276] 1029 08:52:18.432325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.432588 (+ 263us) service_pool.cc:225] Handling call
1029 08:52:19.901236 (+1468648us) ts_tablet_manager.cc:1151] Deleting tablet 391977048e5f4f0b8f62e4d5e3d2946b
1029 08:52:19.901250 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.901382 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1452908}
W20251029 08:52:19.902761 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 391977048e5f4f0b8f62e4d5e3d2946b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 391977048e5f4f0b8f62e4d5e3d2946b already in progress: creating tablet
I20251029 08:52:19.902863 1177 tablet_bootstrap.cc:492] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:19.903016 1177 ts_tablet_manager.cc:1403] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20251029 08:52:19.903357 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 761) took 1470 ms. Trace:
I20251029 08:52:19.903801 1176 rpcz_store.cc:276] 1029 08:52:18.432478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.432806 (+ 328us) service_pool.cc:225] Handling call
1029 08:52:19.903224 (+1470418us) ts_tablet_manager.cc:1151] Deleting tablet 61e96f14b6c74450828ef50148ab7ac5
1029 08:52:19.903235 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.903338 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454770}
W20251029 08:52:19.904554 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:19.904613 1177 raft_consensus.cc:359] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.904852 1177 raft_consensus.cc:385] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.904937 1177 raft_consensus.cc:740] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.905262 1177 consensus_queue.cc:260] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.905392 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 762) took 1469 ms. Trace:
I20251029 08:52:19.905870 1176 rpcz_store.cc:276] 1029 08:52:18.435678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.435732 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:19.905273 (+1469541us) ts_tablet_manager.cc:1151] Deleting tablet 2f36641b7086430a89380056e04b8aaf
1029 08:52:19.905284 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.905374 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456753}
W20251029 08:52:19.906699 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2f36641b7086430a89380056e04b8aaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f36641b7086430a89380056e04b8aaf already in progress: creating tablet
I20251029 08:52:19.907125 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 763) took 1468 ms. Trace:
I20251029 08:52:19.907490 1176 rpcz_store.cc:276] 1029 08:52:18.439992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.440088 (+ 96us) service_pool.cc:225] Handling call
1029 08:52:19.907020 (+1466932us) ts_tablet_manager.cc:1151] Deleting tablet 65eaf12e45c54bd1b0aedc647dc90436
1029 08:52:19.907030 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.907109 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1458452}
W20251029 08:52:19.908159 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 65eaf12e45c54bd1b0aedc647dc90436 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65eaf12e45c54bd1b0aedc647dc90436 already in progress: creating tablet
I20251029 08:52:19.908571 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 764) took 1445 ms. Trace:
I20251029 08:52:19.909018 1183 tablet_bootstrap.cc:492] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.909163 1183 ts_tablet_manager.cc:1403] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.317s user 0.075s sys 0.149s
I20251029 08:52:19.908943 1176 rpcz_store.cc:276] 1029 08:52:18.463621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.463733 (+ 112us) service_pool.cc:225] Handling call
1029 08:52:19.908468 (+1444735us) ts_tablet_manager.cc:1151] Deleting tablet 61e96f14b6c74450828ef50148ab7ac5
1029 08:52:19.908477 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.908555 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444485}
W20251029 08:52:19.909742 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:19.910092 1176 tablet_replica.cc:333] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.910019 1183 raft_consensus.cc:359] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.910219 1183 raft_consensus.cc:385] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.910282 1183 raft_consensus.cc:740] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.910576 1176 raft_consensus.cc:2243] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.910514 1183 consensus_queue.cc:260] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.911036 1183 ts_tablet_manager.cc:1434] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:19.911032 1176 raft_consensus.cc:2272] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.911387 1183 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.911981 1176 ts_tablet_manager.cc:1916] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.913141 1183 tablet_bootstrap.cc:654] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.924335 1176 ts_tablet_manager.cc:1929] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.924583 1176 log.cc:1199] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:19.924911 1183 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.925043 1176 ts_tablet_manager.cc:1950] T fe384ca428974ac5ba78e6506a1a8832 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:19.925074 1183 ts_tablet_manager.cc:1403] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251029 08:52:19.925935 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 776) took 1180 ms. Trace:
I20251029 08:52:19.925904 1183 raft_consensus.cc:359] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.926105 1183 raft_consensus.cc:385] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.926160 1183 raft_consensus.cc:740] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.926388 1183 consensus_queue.cc:260] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:19.926744 1178 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.926867 1178 ts_tablet_manager.cc:1403] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.126s user 0.021s sys 0.027s
I20251029 08:52:19.927016 1183 ts_tablet_manager.cc:1434] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 08:52:19.927286 1183 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.927570 1178 raft_consensus.cc:359] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.927727 1178 raft_consensus.cc:385] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.927778 1178 raft_consensus.cc:740] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.928244 1178 consensus_queue.cc:260] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.929028 1183 tablet_bootstrap.cc:654] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.929446 1178 ts_tablet_manager.cc:1434] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:19.929581 1176 rpcz_store.cc:276] 1029 08:52:18.745917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.746139 (+ 222us) service_pool.cc:225] Handling call
1029 08:52:19.910046 (+1163907us) ts_tablet_manager.cc:1151] Deleting tablet fe384ca428974ac5ba78e6506a1a8832
1029 08:52:19.910056 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.916688 (+ 6632us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.923957 (+ 7269us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.925905 (+ 1948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163663}
I20251029 08:52:19.930267 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet fe384ca428974ac5ba78e6506a1a8832 (table default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]) successfully deleted
I20251029 08:52:19.930948 1178 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.932337 1178 tablet_bootstrap.cc:654] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.932909 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 777) took 1186 ms. Trace:
I20251029 08:52:19.933051 1176 rpcz_store.cc:276] 1029 08:52:18.746081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.746420 (+ 339us) service_pool.cc:225] Handling call
1029 08:52:19.932760 (+1186340us) ts_tablet_manager.cc:1151] Deleting tablet 9a603e0ef22e48f18263ece15b799c0c
1029 08:52:19.932772 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.932889 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186144}
W20251029 08:52:19.933697 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9a603e0ef22e48f18263ece15b799c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a603e0ef22e48f18263ece15b799c0c already in progress: creating tablet
I20251029 08:52:19.934201 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 778) took 1143 ms. Trace:
I20251029 08:52:19.934329 1176 rpcz_store.cc:276] 1029 08:52:18.790332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.790423 (+ 91us) service_pool.cc:225] Handling call
1029 08:52:19.934076 (+1143653us) ts_tablet_manager.cc:1151] Deleting tablet 85df43bf781c421688f163331de0eb2b
1029 08:52:19.934085 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.934183 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143392}
W20251029 08:52:19.934916 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 85df43bf781c421688f163331de0eb2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85df43bf781c421688f163331de0eb2b already in progress: creating tablet
I20251029 08:52:19.935458 1176 tablet_replica.cc:333] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:19.935726 1176 raft_consensus.cc:2243] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.935981 1176 raft_consensus.cc:2272] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.936740 1176 ts_tablet_manager.cc:1916] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.938781 2165 raft_consensus.cc:493] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.938910 2165 raft_consensus.cc:515] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.939527 2165 leader_election.cc:290] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.940912 1177 ts_tablet_manager.cc:1434] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251029 08:52:19.941301 1177 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:19.941403 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dd62833c9d548adabbbf71856a4b181" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.941979 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dd62833c9d548adabbbf71856a4b181" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:19.942812 1177 tablet_bootstrap.cc:654] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:19.944245 718 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:19.944939 720 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:19.945039 720 leader_election.cc:304] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:19.949904 2165 raft_consensus.cc:2749] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:19.955677 2156 raft_consensus.cc:493] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.955852 2156 raft_consensus.cc:515] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.956457 2156 leader_election.cc:290] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:19.958247 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e626994e96d54c0d913fdedc85fd1155" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:19.958487 1070 raft_consensus.cc:2468] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:19.959053 861 leader_election.cc:304] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:19.959508 2156 raft_consensus.cc:2804] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:19.959606 2156 raft_consensus.cc:493] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:19.959678 2156 raft_consensus.cc:3060] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:19.960812 861 leader_election.cc:343] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:19.962510 2156 raft_consensus.cc:515] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.963285 2156 leader_election.cc:290] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:19.963866 861 leader_election.cc:343] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:19.964323 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e626994e96d54c0d913fdedc85fd1155" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:19.964496 1070 raft_consensus.cc:3060] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:19.967481 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087] alter complete (version 1)
I20251029 08:52:19.967748 1070 raft_consensus.cc:2468] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:19.968039 1179 ts_tablet_manager.cc:1929] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:19.968151 1179 log.cc:1199] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7c3e50018f414f4b88e0e5d90cb7c925
I20251029 08:52:19.968189 861 leader_election.cc:304] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:19.968357 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a013257089774ba8b1f535e5f94ac1bb
I20251029 08:52:19.968461 2156 raft_consensus.cc:2804] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:19.968580 2156 raft_consensus.cc:697] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:19.968881 593 catalog_manager.cc:6719] ReplaceTablet: tablet a013257089774ba8b1f535e5f94ac1bb deleted and replaced by tablet bb284e3ca1254697830c0208a9cb7574
I20251029 08:52:19.969236 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_962836b469ba4947acfc9e22d06e89e0" } new_table_name: "default.Table_8364a95e36094945a4d55d737dc1e117" modify_external_catalogs: true
I20251029 08:52:19.969460 887 tablet_service.cc:1552] Processing DeleteTablet for tablet a013257089774ba8b1f535e5f94ac1bb with delete_type TABLET_DATA_DELETED (replaced by tablet bb284e3ca1254697830c0208a9cb7574) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.970144 2156 consensus_queue.cc:237] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:19.970495 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a013257089774ba8b1f535e5f94ac1bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a013257089774ba8b1f535e5f94ac1bb already in progress: creating tablet
I20251029 08:52:19.970698 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ff1b7181e52041288c45b053124bd844
I20251029 08:52:19.971176 588 catalog_manager.cc:6719] ReplaceTablet: tablet ff1b7181e52041288c45b053124bd844 deleted and replaced by tablet 5c02b6343368482e88604ed320147762
I20251029 08:52:19.971475 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ff1b7181e52041288c45b053124bd844 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c02b6343368482e88604ed320147762) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.971645 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_849532564a7c439cb059763db74ef2d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:19.972047 887 tablet_service.cc:1552] Processing DeleteTablet for tablet ff1b7181e52041288c45b053124bd844 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c02b6343368482e88604ed320147762) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:19.972218 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849532564a7c439cb059763db74ef2d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:19.972702 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ff1b7181e52041288c45b053124bd844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b7181e52041288c45b053124bd844 already in progress: creating tablet
I20251029 08:52:19.973649 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a013257089774ba8b1f535e5f94ac1bb with delete_type TABLET_DATA_DELETED (replaced by tablet bb284e3ca1254697830c0208a9cb7574) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.974031 1183 tablet_bootstrap.cc:492] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:19.974072 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8a9f0b579ebd4d2db813459611d7f9bf
I20251029 08:52:19.974179 1183 ts_tablet_manager.cc:1403] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.008s
I20251029 08:52:19.974992 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9c4c2744d70f4d1a9e52b9604edcb436" } modify_external_catalogs: true
I20251029 08:52:19.975037 1183 raft_consensus.cc:359] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.975215 1183 raft_consensus.cc:385] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.975193 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9c4c2744d70f4d1a9e52b9604edcb436" } modify_external_catalogs: true
I20251029 08:52:19.975275 1183 raft_consensus.cc:740] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:19.975431 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.975490 1183 consensus_queue.cc:260] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:19.975657 1974 tablet_replica.cc:333] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:19.975823 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 939d7f6b895d460296eb132087ea65af
I20251029 08:52:19.975837 1974 raft_consensus.cc:2243] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.976034 1974 raft_consensus.cc:2272] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.976291 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a213a91ac12e46ca9bc15e3bb9d39a97
I20251029 08:52:19.976576 1974 ts_tablet_manager.cc:1916] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.977079 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cb859e39a393451a8ec8ff23936fc7ee" } new_table_name: "default.Table_5a446210f03f460b98bbab03e6a98948" modify_external_catalogs: true
I20251029 08:52:19.977442 1179 ts_tablet_manager.cc:1950] T 7c3e50018f414f4b88e0e5d90cb7c925 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:19.977761 1183 ts_tablet_manager.cc:1434] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:19.978029 1183 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:19.978247 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 740) took 1487 ms. Trace:
I20251029 08:52:19.978341 1179 rpcz_store.cc:276] 1029 08:52:18.491078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.491121 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:19.821369 (+1330248us) ts_tablet_manager.cc:1151] Deleting tablet 7c3e50018f414f4b88e0e5d90cb7c925
1029 08:52:19.821379 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.932678 (+111299us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.967901 (+ 35223us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:19.978227 (+ 10326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1330073}
I20251029 08:52:19.979151 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b2ca7c25488a477fbe498883e704971c" } new_table_name: "default.Table_fe464f17db3547f393d347ee7c8643c2" modify_external_catalogs: true
I20251029 08:52:19.979624 1183 tablet_bootstrap.cc:654] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:19.980585 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7c3e50018f414f4b88e0e5d90cb7c925 (table default.Table_f6ce198ce16b46249069916455fe0165 [id=a9001dc841b94f4ca65db7ca69fb01fe]) successfully deleted
I20251029 08:52:19.980839 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.980868 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:19.981278 745 tablet_service.cc:1552] Processing DeleteTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.981937 887 tablet_service.cc:1552] Processing DeleteTablet for tablet a213a91ac12e46ca9bc15e3bb9d39a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:19.983705 1179 tablet_replica.cc:333] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:19.983883 1179 raft_consensus.cc:2243] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:19.984900 1179 raft_consensus.cc:2272] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:19.985978 1974 ts_tablet_manager.cc:1929] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:19.986104 1974 log.cc:1199] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/8a9f0b579ebd4d2db813459611d7f9bf
I20251029 08:52:19.986119 1179 ts_tablet_manager.cc:1916] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:19.986519 1974 ts_tablet_manager.cc:1950] T 8a9f0b579ebd4d2db813459611d7f9bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:19.987982 1178 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:19.988081 594 master_service.cc:954] ReplaceTablet: received request to replace tablet a8c4d57a59684723bfd27c7dda302ef3 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.988174 1178 ts_tablet_manager.cc:1403] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.007s
I20251029 08:52:19.988638 594 master_service.cc:954] ReplaceTablet: received request to replace tablet d9cabbbac8254a949750244e1d6991eb from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.989130 1178 raft_consensus.cc:359] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.989351 1178 raft_consensus.cc:385] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:19.989411 1178 raft_consensus.cc:740] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:19.989859 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 consensus state has no leader
I20251029 08:52:19.990106 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:19.990360 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a9f0b579ebd4d2db813459611d7f9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:19.990598 2165 raft_consensus.cc:493] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:19.990744 2165 raft_consensus.cc:515] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.991214 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]) successfully deleted
I20251029 08:52:19.991532 2165 leader_election.cc:290] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:19.993273 1178 consensus_queue.cc:260] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:19.993816 1178 ts_tablet_manager.cc:1434] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20251029 08:52:19.994683 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a213a91ac12e46ca9bc15e3bb9d39a97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a213a91ac12e46ca9bc15e3bb9d39a97 already in progress: creating tablet
I20251029 08:52:19.995219 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 939d7f6b895d460296eb132087ea65af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:19.995961 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 939d7f6b895d460296eb132087ea65af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 939d7f6b895d460296eb132087ea65af already in progress: creating tablet
I20251029 08:52:19.998924 1178 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:19.999646 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:19.999992 998 consensus_peers.cc:597] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5385fef404a4463595074f0c8ff822d7. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:20.000555 1178 tablet_bootstrap.cc:654] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:20.001438 718 leader_election.cc:343] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.002959 1176 ts_tablet_manager.cc:1929] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.003072 1176 log.cc:1199] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:20.003361 1176 ts_tablet_manager.cc:1950] T de78ea608c904aeea7f7145b9c80f45b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.003957 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_837fc07aec184f858821c9c37cb761cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.004128 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 779) took 1202 ms. Trace:
I20251029 08:52:20.004200 1176 rpcz_store.cc:276] 1029 08:52:18.801210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.801268 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:19.935406 (+1134138us) ts_tablet_manager.cc:1151] Deleting tablet de78ea608c904aeea7f7145b9c80f45b
1029 08:52:19.935416 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.961693 (+ 26277us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.002833 (+ 41140us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.004109 (+ 1276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133934}
I20251029 08:52:20.004601 1176 tablet_replica.cc:333] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:20.004595 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_837fc07aec184f858821c9c37cb761cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.004786 1176 raft_consensus.cc:2243] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.004920 1176 raft_consensus.cc:2272] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.005406 1179 ts_tablet_manager.cc:1929] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:20.005417 1176 ts_tablet_manager.cc:1916] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.005511 1179 log.cc:1199] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/072cf27b0743488c8166e03b5e61c545
I20251029 08:52:20.005928 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet de78ea608c904aeea7f7145b9c80f45b (table default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]) successfully deleted
I20251029 08:52:20.007972 887 tablet_service.cc:1552] Processing DeleteTablet for tablet e626994e96d54c0d913fdedc85fd1155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.008481 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 939d7f6b895d460296eb132087ea65af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.008519 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:20.009176 720 leader_election.cc:343] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.009330 720 leader_election.cc:304] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.009611 2038 raft_consensus.cc:2749] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.008317 1974 tablet_replica.cc:333] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:20.016605 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b0b3b6fa45044c3fb83823ed4f233735
I20251029 08:52:20.021046 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.021128 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.021749 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b0b3b6fa45044c3fb83823ed4f233735 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.025810 1176 ts_tablet_manager.cc:1929] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.026437 1176 log.cc:1199] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aafa846cb2144cec9ce4b28e61d91961
I20251029 08:52:20.027105 1176 ts_tablet_manager.cc:1950] T aafa846cb2144cec9ce4b28e61d91961 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.028698 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 787d6d7f1870450fa21cb9cba34e72fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.028774 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c706343d44a4807b1ea60c1b1e1ea49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.029141 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.029318 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_554113ad7eea4ccfa7b3b4af7f145923"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.029985 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_554113ad7eea4ccfa7b3b4af7f145923 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.030042 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 780) took 1228 ms. Trace:
I20251029 08:52:20.030556 1176 rpcz_store.cc:276] 1029 08:52:18.801941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.801987 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:20.004551 (+1202564us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:20.004561 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.018216 (+ 13655us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.025688 (+ 7472us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.030020 (+ 4332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202389}
W20251029 08:52:20.031097 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ed4f048f1ae4d70aa1b88116a57e2bd has no consensus state
I20251029 08:52:20.031342 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet aafa846cb2144cec9ce4b28e61d91961 (table default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]) successfully deleted
I20251029 08:52:20.031500 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_691d182612d74f709b0e5d106e4a5e11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.031693 1176 tablet_replica.cc:333] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.031867 1176 raft_consensus.cc:2243] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.031991 1176 raft_consensus.cc:2272] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.032073 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691d182612d74f709b0e5d106e4a5e11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.033960 1176 ts_tablet_manager.cc:1916] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.034224 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d9cabbbac8254a949750244e1d6991eb
I20251029 08:52:20.034732 594 catalog_manager.cc:6719] ReplaceTablet: tablet d9cabbbac8254a949750244e1d6991eb deleted and replaced by tablet b1f6ee4fe18d41bf8573dbd8f02f49dd
I20251029 08:52:20.035063 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5cd8400b299442c2bcddea0bd1b91697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.035578 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cd8400b299442c2bcddea0bd1b91697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.037406 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d9cabbbac8254a949750244e1d6991eb with delete_type TABLET_DATA_DELETED (replaced by tablet b1f6ee4fe18d41bf8573dbd8f02f49dd) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:20.037462 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 consensus state has no leader
I20251029 08:52:20.038010 893 tablet_service.cc:1552] Processing DeleteTablet for tablet d9cabbbac8254a949750244e1d6991eb with delete_type TABLET_DATA_DELETED (replaced by tablet b1f6ee4fe18d41bf8573dbd8f02f49dd) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:20.038746 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72d0540040614513a4518ac983d14a94 has no consensus state
W20251029 08:52:20.039556 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bfa85dfcdad46d3a7cbc7150c3b625f consensus state has no leader
I20251029 08:52:20.041903 2016 consensus_queue.cc:1048] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:20.044354 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d9cabbbac8254a949750244e1d6991eb with delete_type TABLET_DATA_DELETED (replaced by tablet b1f6ee4fe18d41bf8573dbd8f02f49dd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.051033 1176 ts_tablet_manager.cc:1929] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.053334 1176 log.cc:1199] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:20.053949 1176 ts_tablet_manager.cc:1950] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.055351 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 781) took 1252 ms. Trace:
I20251029 08:52:20.056089 1176 rpcz_store.cc:276] 1029 08:52:18.802556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.802606 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:20.031641 (+1229035us) ts_tablet_manager.cc:1151] Deleting tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c
1029 08:52:20.031652 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.041162 (+ 9510us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.050909 (+ 9747us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.055333 (+ 4424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228816}
I20251029 08:52:20.056931 1176 tablet_replica.cc:333] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.057325 1176 raft_consensus.cc:2243] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.057711 1176 raft_consensus.cc:2272] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.058952 1176 ts_tablet_manager.cc:1916] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.059564 1179 ts_tablet_manager.cc:1950] T 072cf27b0743488c8166e03b5e61c545 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.062340 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 741) took 1571 ms. Trace:
I20251029 08:52:20.059924 1178 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.064895 1178 ts_tablet_manager.cc:1403] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.011s
I20251029 08:52:20.065140 1179 rpcz_store.cc:276] 1029 08:52:18.491310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.491354 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:19.983467 (+1492113us) ts_tablet_manager.cc:1151] Deleting tablet 072cf27b0743488c8166e03b5e61c545
1029 08:52:19.983666 (+ 199us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:19.989335 (+ 5669us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.005302 (+ 15967us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.062307 (+ 57005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491981}
I20251029 08:52:20.065994 1178 raft_consensus.cc:359] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.066329 1178 raft_consensus.cc:385] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.066443 1178 raft_consensus.cc:740] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.066890 1178 consensus_queue.cc:260] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.067003 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 742) took 1575 ms. Trace:
I20251029 08:52:20.067090 1179 rpcz_store.cc:276] 1029 08:52:18.491399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.496122 (+ 4723us) service_pool.cc:225] Handling call
1029 08:52:20.066845 (+1570723us) ts_tablet_manager.cc:1151] Deleting tablet 61e96f14b6c74450828ef50148ab7ac5
1029 08:52:20.066858 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.066980 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1570556}
I20251029 08:52:20.068903 1178 ts_tablet_manager.cc:1434] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:20.069579 1178 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.069800 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 754) took 1315 ms. Trace:
I20251029 08:52:20.070883 1179 rpcz_store.cc:276] 1029 08:52:18.754439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.754510 (+ 71us) service_pool.cc:225] Handling call
1029 08:52:20.069655 (+1315145us) ts_tablet_manager.cc:1151] Deleting tablet 9a603e0ef22e48f18263ece15b799c0c
1029 08:52:20.069667 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.069778 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314892}
I20251029 08:52:20.071885 1178 tablet_bootstrap.cc:654] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.071893 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet ff1b7181e52041288c45b053124bd844 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c02b6343368482e88604ed320147762) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.072237 1179 tablet_replica.cc:333] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.073000 1179 raft_consensus.cc:2243] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:20.074491 1179 pending_rounds.cc:70] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:20.075397 1179 pending_rounds.cc:77] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037638081912832 op_type: NO_OP noop_request { }
I20251029 08:52:20.075965 1179 raft_consensus.cc:2889] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:20.075941 2188 tablet.cc:1722] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.076581 1179 raft_consensus.cc:2272] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.077350 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (table default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]) successfully deleted
I20251029 08:52:20.077633 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a4078e8eb124424399751be94d3a19b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.077862 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 072cf27b0743488c8166e03b5e61c545 (table default.Table_e6415f97a9d64fc996f4e571e61aba62 [id=2f35fa7c4b344c9f940c8e7eea83e13a]) successfully deleted
I20251029 08:52:20.078131 1179 ts_tablet_manager.cc:1916] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.078423 1176 ts_tablet_manager.cc:1929] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.078697 1176 log.cc:1199] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:20.079093 1176 ts_tablet_manager.cc:1950] T 24ab9df6f768410eb4f66cffdcd68df2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.080940 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 782) took 1272 ms. Trace:
I20251029 08:52:20.081799 1176 rpcz_store.cc:276] 1029 08:52:18.808725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.808787 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:20.056880 (+1248093us) ts_tablet_manager.cc:1151] Deleting tablet 24ab9df6f768410eb4f66cffdcd68df2
1029 08:52:20.056890 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.063606 (+ 6716us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.075990 (+ 12384us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.080920 (+ 4930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1247891}
I20251029 08:52:20.081588 1178 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.082195 1178 ts_tablet_manager.cc:1403] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251029 08:52:20.082893 1178 raft_consensus.cc:359] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.083101 1178 raft_consensus.cc:385] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.083190 1178 raft_consensus.cc:740] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.083591 1178 consensus_queue.cc:260] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:20.084201 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 61e96f14b6c74450828ef50148ab7ac5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e96f14b6c74450828ef50148ab7ac5 already in progress: creating tablet
I20251029 08:52:20.084514 1178 ts_tablet_manager.cc:1434] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251029 08:52:20.084640 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9a603e0ef22e48f18263ece15b799c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a603e0ef22e48f18263ece15b799c0c already in progress: creating tablet
I20251029 08:52:20.084870 1178 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.086205 1178 tablet_bootstrap.cc:654] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.087126 1176 tablet_replica.cc:333] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.087782 1176 raft_consensus.cc:2243] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.088662 1176 raft_consensus.cc:2272] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.089253 1176 ts_tablet_manager.cc:1916] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.096166 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 939d7f6b895d460296eb132087ea65af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.099889 1176 ts_tablet_manager.cc:1929] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.100144 1176 log.cc:1199] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bda3e143699b41b6bba308c83bbb3e0d
I20251029 08:52:20.100495 1176 ts_tablet_manager.cc:1950] T bda3e143699b41b6bba308c83bbb3e0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.101286 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 788) took 1003 ms. Trace:
I20251029 08:52:20.101425 1176 rpcz_store.cc:276] 1029 08:52:19.097544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.098000 (+ 456us) service_pool.cc:225] Handling call
1029 08:52:20.087079 (+989079us) ts_tablet_manager.cc:1151] Deleting tablet bda3e143699b41b6bba308c83bbb3e0d
1029 08:52:20.087088 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.093870 (+ 6782us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.099800 (+ 5930us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.101265 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988890}
I20251029 08:52:20.102519 1176 tablet_replica.cc:333] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.102777 1176 raft_consensus.cc:2243] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.102977 1176 raft_consensus.cc:2272] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.103493 1176 ts_tablet_manager.cc:1916] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.104892 1178 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.105050 1178 ts_tablet_manager.cc:1403] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20251029 08:52:20.105780 2044 raft_consensus.cc:493] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.105758 1178 raft_consensus.cc:359] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.105917 1178 raft_consensus.cc:385] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.105965 1178 raft_consensus.cc:740] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.105973 2044 raft_consensus.cc:515] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.106439 1178 consensus_queue.cc:260] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.106765 2044 leader_election.cc:290] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.107165 1178 ts_tablet_manager.cc:1434] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.107722 1178 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:20.108088 998 leader_election.cc:343] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2c706343d44a4807b1ea60c1b1e1ea49
W20251029 08:52:20.108667 996 leader_election.cc:343] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 2c706343d44a4807b1ea60c1b1e1ea49
I20251029 08:52:20.108772 996 leader_election.cc:304] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.109534 2044 raft_consensus.cc:2749] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:20.109512 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4078e8eb124424399751be94d3a19b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.109954 1178 tablet_bootstrap.cc:654] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.109863 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1938936fb1a5483c9d118a57d88d283b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.110553 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1938936fb1a5483c9d118a57d88d283b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.110828 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_626c2d6e7b6b4dea95389dd85a113864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.113446 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_626c2d6e7b6b4dea95389dd85a113864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.117746 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_656e3c6b753e4109bc607af874503567" } new_table_name: "default.Table_893d4688053844dd97c9140f9abb0667" modify_external_catalogs: true
I20251029 08:52:20.118319 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 24ab9df6f768410eb4f66cffdcd68df2 (table default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]) successfully deleted
I20251029 08:52:20.118582 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bda3e143699b41b6bba308c83bbb3e0d (table default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]) successfully deleted
W20251029 08:52:20.119055 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 already in progress: creating tablet
W20251029 08:52:20.120301 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cf09e8fd3d2b43318fa37d922c8b826f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf09e8fd3d2b43318fa37d922c8b826f already in progress: creating tablet
I20251029 08:52:20.118991 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3676333781d94b0d80010190fd318da8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.121928 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3676333781d94b0d80010190fd318da8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.134045 1176 ts_tablet_manager.cc:1929] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.137079 1179 ts_tablet_manager.cc:1929] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:20.138614 1179 log.cc:1199] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/5385fef404a4463595074f0c8ff822d7
I20251029 08:52:20.139832 1179 ts_tablet_manager.cc:1950] T 5385fef404a4463595074f0c8ff822d7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.139808 1178 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.141665 1178 ts_tablet_manager.cc:1403] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20251029 08:52:20.142446 1178 raft_consensus.cc:359] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.142822 1178 raft_consensus.cc:385] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.142916 1178 raft_consensus.cc:740] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:20.143015 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8f8a87bf57e4b1e9d08336c544b472b consensus state has no leader
I20251029 08:52:20.143189 1178 consensus_queue.cc:260] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.143909 1178 ts_tablet_manager.cc:1434] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.144702 1178 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:20.146529 2066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251029 08:52:20.147037 1178 tablet_bootstrap.cc:654] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.147194 1176 log.cc:1199] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/75e1f0cdfbc34c6f85f2ad545eb33c0d
I20251029 08:52:20.148326 1176 ts_tablet_manager.cc:1950] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:20.148595 2066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 08:52:20.149093 2066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 08:52:20.150481 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 73294.7us
I20251029 08:52:20.151417 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (table default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]) successfully deleted
I20251029 08:52:20.152856 1176 tablet_replica.cc:333] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.153095 1176 raft_consensus.cc:2243] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.153523 1176 raft_consensus.cc:2272] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.155517 1176 ts_tablet_manager.cc:1916] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:20.165798 861 consensus_peers.cc:597] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:20.166664 718 consensus_peers.cc:597] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a9f0b579ebd4d2db813459611d7f9bf. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:20.172009 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 755) took 1355 ms. Trace:
I20251029 08:52:20.172262 1179 rpcz_store.cc:276] 1029 08:52:18.816308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.816414 (+ 106us) service_pool.cc:225] Handling call
1029 08:52:20.072185 (+1255771us) ts_tablet_manager.cc:1151] Deleting tablet 5385fef404a4463595074f0c8ff822d7
1029 08:52:20.072195 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.098517 (+ 26322us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.136920 (+ 38403us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.171978 (+ 35058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255520}
I20251029 08:52:20.173578 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 758) took 1357 ms. Trace:
I20251029 08:52:20.156585 2144 tablet.cc:1722] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.145535 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet a013257089774ba8b1f535e5f94ac1bb with delete_type TABLET_DATA_DELETED (replaced by tablet bb284e3ca1254697830c0208a9cb7574) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:20.175648 2066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20251029 08:52:20.175890 2165 raft_consensus.cc:493] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.176119 2165 raft_consensus.cc:515] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.176940 2165 leader_election.cc:290] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.177951 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39fc9612689b46d498d6092bd552da4e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.178017 2165 raft_consensus.cc:493] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.178182 2165 raft_consensus.cc:515] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:20.178825 2066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20251029 08:52:20.178917 2165 leader_election.cc:290] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:20.179221 2066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 08:52:20.179508 2066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 08:52:20.179869 1178 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.180096 1178 ts_tablet_manager.cc:1403] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20251029 08:52:20.181047 1178 raft_consensus.cc:359] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.181236 1178 raft_consensus.cc:385] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.181330 1178 raft_consensus.cc:740] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.181699 1178 consensus_queue.cc:260] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.182474 1178 ts_tablet_manager.cc:1434] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:20.182783 1178 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.184255 1178 tablet_bootstrap.cc:654] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.185153 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd0c43367a8c47f390cc61d2764d61e4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:20.184602 718 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:20.187382 718 leader_election.cc:343] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.188980 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5df6707d7a754de3b61f32cf68786b2e" } modify_external_catalogs: true
I20251029 08:52:20.189162 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5df6707d7a754de3b61f32cf68786b2e" } modify_external_catalogs: true
I20251029 08:52:20.189764 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e857915b93d94731a6672caf050b7807" } modify_external_catalogs: true
I20251029 08:52:20.189916 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e857915b93d94731a6672caf050b7807" } modify_external_catalogs: true
I20251029 08:52:20.193203 593 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab] alter complete (version 2)
I20251029 08:52:20.195921 2187 tablet.cc:1722] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.196112 1974 raft_consensus.cc:2243] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:20.196740 1974 pending_rounds.cc:70] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:20.196797 1974 pending_rounds.cc:77] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037642120331264 op_type: NO_OP noop_request { }
I20251029 08:52:20.196887 1974 raft_consensus.cc:2889] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:20.196974 1974 raft_consensus.cc:2272] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.202739 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a513e19699c24ff58a3d6a6fcae4c96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.203352 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a513e19699c24ff58a3d6a6fcae4c96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.204679 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c253bd6532c8472295cf1d50dfc59d42" } new_table_name: "default.Table_936021e20d444b0e9067bb236b20bae2" modify_external_catalogs: true
I20251029 08:52:20.210281 1176 ts_tablet_manager.cc:1929] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:20.210484 1974 ts_tablet_manager.cc:1916] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.216252 1176 log.cc:1199] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c4e1b6b6359b437aa295b473fa42dce1
I20251029 08:52:20.217223 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ab4c150f8dd4dfdba46c3555d68e653" } modify_external_catalogs: true
I20251029 08:52:20.217448 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ab4c150f8dd4dfdba46c3555d68e653" } modify_external_catalogs: true
W20251029 08:52:20.217973 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41622daeb1e0419fa88cc26c0405c2db consensus state has no leader
I20251029 08:52:20.218663 1176 ts_tablet_manager.cc:1950] T c4e1b6b6359b437aa295b473fa42dce1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.219113 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:20.219717 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:20.219812 1974 ts_tablet_manager.cc:1929] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:20.219938 1974 log.cc:1199] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:20.220433 1974 ts_tablet_manager.cc:1950] T e626994e96d54c0d913fdedc85fd1155 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:20.221378 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e626994e96d54c0d913fdedc85fd1155 (table default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]) successfully deleted
W20251029 08:52:20.221697 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b0b3b6fa45044c3fb83823ed4f233735 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b3b6fa45044c3fb83823ed4f233735 already in progress: creating tablet
W20251029 08:52:20.221937 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d9cabbbac8254a949750244e1d6991eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9cabbbac8254a949750244e1d6991eb already in progress: creating tablet
I20251029 08:52:20.222008 1179 rpcz_store.cc:276] 1029 08:52:18.816566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.816716 (+ 150us) service_pool.cc:225] Handling call
1029 08:52:20.172892 (+1356176us) ts_tablet_manager.cc:1151] Deleting tablet 85df43bf781c421688f163331de0eb2b
1029 08:52:20.172902 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.173551 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355983}
I20251029 08:52:20.222451 1179 tablet_replica.cc:333] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.222970 1179 raft_consensus.cc:2243] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.223210 1179 raft_consensus.cc:2272] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.222721 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 114056us
I20251029 08:52:20.224097 1178 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.224246 1178 ts_tablet_manager.cc:1403] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20251029 08:52:20.224885 1183 tablet_bootstrap.cc:492] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.225014 1183 ts_tablet_manager.cc:1403] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.247s user 0.033s sys 0.033s
I20251029 08:52:20.225633 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 5385fef404a4463595074f0c8ff822d7 (table default.Table_58fa7040e0e2467588dd8b5a0a2eba4b [id=2cfb2b85354d43e7a203d35a4448346d]) successfully deleted
I20251029 08:52:20.225592 1178 raft_consensus.cc:359] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.225741 1177 tablet_bootstrap.cc:492] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.225775 1178 raft_consensus.cc:385] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.225826 1178 raft_consensus.cc:740] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.225744 1183 raft_consensus.cc:359] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.225864 1177 ts_tablet_manager.cc:1403] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.285s user 0.011s sys 0.020s
I20251029 08:52:20.225900 1183 raft_consensus.cc:385] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.225957 1183 raft_consensus.cc:740] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.226087 1178 consensus_queue.cc:260] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.226184 1183 consensus_queue.cc:260] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.226861 1183 ts_tablet_manager.cc:1434] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.227479 1178 ts_tablet_manager.cc:1434] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:20.227742 1183 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.227247 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_76bc2e4759364fe488fa32a2afa440a0" } modify_external_catalogs: true
I20251029 08:52:20.228065 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_76bc2e4759364fe488fa32a2afa440a0" } modify_external_catalogs: true
I20251029 08:52:20.228395 1178 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.228662 1177 raft_consensus.cc:359] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.228832 1177 raft_consensus.cc:385] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.228888 1177 raft_consensus.cc:740] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.229081 1177 consensus_queue.cc:260] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.229483 1183 tablet_bootstrap.cc:654] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.229888 1178 tablet_bootstrap.cc:654] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.230170 1177 ts_tablet_manager.cc:1434] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:20.232177 1177 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:20.233695 1177 tablet_bootstrap.cc:654] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.234020 2197 raft_consensus.cc:493] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.234151 2197 raft_consensus.cc:515] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.234777 2197 leader_election.cc:290] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.235092 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5cd8400b299442c2bcddea0bd1b91697" } new_table_name: "default.Table_47a0b540f1834bd8bd754a77e504c6e5" modify_external_catalogs: true
I20251029 08:52:20.220834 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 794) took 1001 ms. Trace:
W20251029 08:52:20.240139 998 leader_election.cc:343] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:20.240360 1176 rpcz_store.cc:276] 1029 08:52:19.219674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.247466 (+ 27792us) service_pool.cc:225] Handling call
1029 08:52:20.152790 (+905324us) ts_tablet_manager.cc:1151] Deleting tablet c4e1b6b6359b437aa295b473fa42dce1
1029 08:52:20.152803 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.174821 (+ 22018us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.210145 (+ 35324us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.220809 (+ 10664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904603}
I20251029 08:52:20.241641 1176 tablet_replica.cc:333] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.241902 1176 raft_consensus.cc:2243] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.242094 1176 raft_consensus.cc:2272] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.242679 1176 ts_tablet_manager.cc:1916] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.245306 2066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40435,127.0.104.61:38825,127.0.104.60:42135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=37745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 08:43:21 UTC on 24a791456cd2
build id 8772
ASAN enabled
W20251029 08:52:20.250962 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b91221923dae416dbf046b555f390d99 has no consensus state
I20251029 08:52:20.251443 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_82984426f7eb4edf9c0cab2e8544d1dc" } new_table_name: "default.Table_61766253e4ff493b98bfa21f283633e7" modify_external_catalogs: true
I20251029 08:52:20.254622 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
I20251029 08:52:20.257782 1176 ts_tablet_manager.cc:1929] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:20.258214 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e14d05e64689423da77d6d60b760cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d05e64689423da77d6d60b760cc06 already in progress: creating tablet
W20251029 08:52:20.258508 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 663d6587a5e14519afbb0f94d691d1b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663d6587a5e14519afbb0f94d691d1b4 already in progress: creating tablet
W20251029 08:52:20.258715 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 47f58be6d7f5492395ca6192f7ce0930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47f58be6d7f5492395ca6192f7ce0930 already in progress: creating tablet
I20251029 08:52:20.259244 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4ff1bcc056504a13864a32f4f16bceb9" } new_table_name: "default.Table_d29aed5fe8f44c47a0f174f5d5fd9d5f" modify_external_catalogs: true
I20251029 08:52:20.260035 1176 log.cc:1199] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:20.260345 1176 ts_tablet_manager.cc:1950] T af38662420d14ac1a7cb5fd03fc1b9d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.262107 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_95ce9269797c4de3b5bd2c1cc2a6f384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.262987 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.264179 1176 tablet_replica.cc:333] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:20.264159 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95ce9269797c4de3b5bd2c1cc2a6f384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.263622 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet af38662420d14ac1a7cb5fd03fc1b9d8 (table default.Table_69a638baaadc493f9439102862bb7607 [id=14f726f4d8104eaf99f5dbb58f9ce22c]) successfully deleted
I20251029 08:52:20.264565 1176 raft_consensus.cc:2243] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:20.264600 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ff1b7181e52041288c45b053124bd844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b7181e52041288c45b053124bd844 already in progress: creating tablet
W20251029 08:52:20.264933 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a013257089774ba8b1f535e5f94ac1bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a013257089774ba8b1f535e5f94ac1bb already in progress: creating tablet
I20251029 08:52:20.265228 1176 raft_consensus.cc:2272] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.265389 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a213a91ac12e46ca9bc15e3bb9d39a97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a213a91ac12e46ca9bc15e3bb9d39a97 already in progress: creating tablet
I20251029 08:52:20.266099 1183 tablet_bootstrap.cc:492] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.266238 1176 ts_tablet_manager.cc:1916] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.266295 1183 ts_tablet_manager.cc:1403] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.001s
I20251029 08:52:20.267416 1183 raft_consensus.cc:359] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.267658 1183 raft_consensus.cc:385] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.267751 1183 raft_consensus.cc:740] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.268011 1183 consensus_queue.cc:260] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.268942 1183 ts_tablet_manager.cc:1434] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.269098 1974 tablet_replica.cc:333] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:20.269454 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6691e1944ae947b49e4431f31ba9ae41 consensus state has no leader
I20251029 08:52:20.269369 1974 raft_consensus.cc:2243] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.270385 1974 raft_consensus.cc:2272] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.271142 2066 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 08:52:20.272388 1974 ts_tablet_manager.cc:1916] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.273643 2066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 08:52:20.274078 1183 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.274138 748 tablet_service.cc:1505] Processing CreateTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac (DEFAULT_TABLE table=default.Table_7c57b7890d1f4c8e8e9801e845e4df11 [id=ceebd69263174829ae64d1d71d7d1f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.274551 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.278299 1183 tablet_bootstrap.cc:654] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.279625 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:20.279659 745 tablet_service.cc:1505] Processing CreateTablet for tablet e8346fa8652c4233a289b539875e0507 (DEFAULT_TABLE table=default.Table_f29176d94379462f980b648a202d8a6a [id=c5d9ae66f1ed4012ad743406a5e42f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.279865 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:20.280073 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8346fa8652c4233a289b539875e0507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.274438 1176 ts_tablet_manager.cc:1929] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:20.281872 1176 log.cc:1199] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8a9f0b579ebd4d2db813459611d7f9bf
I20251029 08:52:20.282274 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:20.282543 758 tablet_service.cc:1505] Processing CreateTablet for tablet 7492195a41664ef69554d8a6cb148e21 (DEFAULT_TABLE table=default.Table_4b68078657014c3790ceed68233e826e [id=15cdc3d5561948b991ed50cefeded756]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.282915 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7492195a41664ef69554d8a6cb148e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.279908 747 tablet_service.cc:1505] Processing CreateTablet for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a (DEFAULT_TABLE table=default.Table_d56c826be1cc4fc1a9a8c42bf71799b1 [id=f4b6717002df46beb4bd34daef2d01ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.283779 1176 ts_tablet_manager.cc:1950] T 8a9f0b579ebd4d2db813459611d7f9bf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.288522 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.288924 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.289347 1176 tablet_replica.cc:333] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.289542 1176 raft_consensus.cc:2243] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.289700 1176 raft_consensus.cc:2272] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.290208 1176 ts_tablet_manager.cc:1916] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.291169 1183 tablet_bootstrap.cc:492] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.291294 1183 ts_tablet_manager.cc:1403] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20251029 08:52:20.292081 1183 raft_consensus.cc:359] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.292308 1183 raft_consensus.cc:385] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.292409 1183 raft_consensus.cc:740] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.292699 1183 consensus_queue.cc:260] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.293001 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.293198 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.293419 757 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.293607 1183 ts_tablet_manager.cc:1434] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.293928 1183 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.296185 1183 tablet_bootstrap.cc:654] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.296877 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]) successfully deleted
W20251029 08:52:20.297135 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b0b3b6fa45044c3fb83823ed4f233735 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b3b6fa45044c3fb83823ed4f233735 already in progress: creating tablet
W20251029 08:52:20.298755 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d9cabbbac8254a949750244e1d6991eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9cabbbac8254a949750244e1d6991eb already in progress: creating tablet
W20251029 08:52:20.299098 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 939d7f6b895d460296eb132087ea65af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 939d7f6b895d460296eb132087ea65af already in progress: creating tablet
I20251029 08:52:20.286051 893 tablet_service.cc:1505] Processing CreateTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac (DEFAULT_TABLE table=default.Table_7c57b7890d1f4c8e8e9801e845e4df11 [id=ceebd69263174829ae64d1d71d7d1f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.303004 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.304169 2197 raft_consensus.cc:493] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.304322 2197 raft_consensus.cc:515] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.304926 2197 leader_election.cc:290] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.305070 1183 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.305235 1183 ts_tablet_manager.cc:1403] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
W20251029 08:52:20.305898 998 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:20.306082 1183 raft_consensus.cc:359] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.306324 1183 raft_consensus.cc:385] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.306419 1183 raft_consensus.cc:740] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.306700 1183 consensus_queue.cc:260] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.307374 2197 raft_consensus.cc:493] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.286688 892 tablet_service.cc:1505] Processing CreateTablet for tablet e8346fa8652c4233a289b539875e0507 (DEFAULT_TABLE table=default.Table_f29176d94379462f980b648a202d8a6a [id=c5d9ae66f1ed4012ad743406a5e42f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.307484 2197 raft_consensus.cc:515] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.307816 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8346fa8652c4233a289b539875e0507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.308001 2197 leader_election.cc:290] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.307332 1183 ts_tablet_manager.cc:1434] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.308710 1183 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:20.309379 998 leader_election.cc:343] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e4f80728a7a04583bada485707beb806
I20251029 08:52:20.310379 1183 tablet_bootstrap.cc:654] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.286877 901 tablet_service.cc:1505] Processing CreateTablet for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a (DEFAULT_TABLE table=default.Table_d56c826be1cc4fc1a9a8c42bf71799b1 [id=f4b6717002df46beb4bd34daef2d01ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.311348 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.312822 1178 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.312951 1178 ts_tablet_manager.cc:1403] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.013s
I20251029 08:52:20.313696 887 tablet_service.cc:1505] Processing CreateTablet for tablet a17142ffce3e4c359e715ef63747c40d (DEFAULT_TABLE table=default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6 [id=150e96b3fa8f4fbd8dd20d8413f3483a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.314025 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17142ffce3e4c359e715ef63747c40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.287038 896 tablet_service.cc:1505] Processing CreateTablet for tablet 7492195a41664ef69554d8a6cb148e21 (DEFAULT_TABLE table=default.Table_4b68078657014c3790ceed68233e826e [id=15cdc3d5561948b991ed50cefeded756]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.316407 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7492195a41664ef69554d8a6cb148e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.319200 1183 tablet_bootstrap.cc:492] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.319329 1183 ts_tablet_manager.cc:1403] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20251029 08:52:20.320091 1183 raft_consensus.cc:359] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.320252 1183 raft_consensus.cc:385] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.320298 1183 raft_consensus.cc:740] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.320544 1183 consensus_queue.cc:260] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.320719 2197 raft_consensus.cc:493] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.320871 2197 raft_consensus.cc:515] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.321141 1183 ts_tablet_manager.cc:1434] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.321409 1183 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.321614 2197 leader_election.cc:290] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.287230 902 tablet_service.cc:1505] Processing CreateTablet for tablet 18002650e1934b659611e0fbb74b4601 (DEFAULT_TABLE table=default.Table_bbcfd7574bc64702be1c06a2105814d1 [id=a54133947aac4cecb8ffd33bbe39fa70]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:20.322741 998 leader_election.cc:343] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fd25eeb89e574297a92d8e978a79d768
I20251029 08:52:20.322858 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18002650e1934b659611e0fbb74b4601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.323086 1183 tablet_bootstrap.cc:654] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.323186 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39fc9612689b46d498d6092bd552da4e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.325992 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd0c43367a8c47f390cc61d2764d61e4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.328325 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2723cdd513ee4e8ca6773b38f785b09e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.331110 745 tablet_service.cc:1505] Processing CreateTablet for tablet 18002650e1934b659611e0fbb74b4601 (DEFAULT_TABLE table=default.Table_bbcfd7574bc64702be1c06a2105814d1 [id=a54133947aac4cecb8ffd33bbe39fa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.331665 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd25eeb89e574297a92d8e978a79d768" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.332125 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18002650e1934b659611e0fbb74b4601. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:20.332952 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 85df43bf781c421688f163331de0eb2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85df43bf781c421688f163331de0eb2b already in progress: creating tablet
W20251029 08:52:20.334131 720 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.334254 720 leader_election.cc:304] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.334656 2165 raft_consensus.cc:2749] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:20.335515 720 leader_election.cc:343] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.335623 720 leader_election.cc:304] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.336732 2165 raft_consensus.cc:2749] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.336766 1178 raft_consensus.cc:359] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.336925 1178 raft_consensus.cc:385] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.336983 1178 raft_consensus.cc:740] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.337162 1178 consensus_queue.cc:260] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.337634 1178 ts_tablet_manager.cc:1434] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251029 08:52:20.337934 1178 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.342818 1176 ts_tablet_manager.cc:1929] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.343060 1176 log.cc:1199] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:20.343540 1176 ts_tablet_manager.cc:1950] T 1dd62833c9d548adabbbf71856a4b181 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.347823 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet c572c68b32d443b2a019e1f4bf2d82fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.349148 758 tablet_service.cc:1505] Processing CreateTablet for tablet a17142ffce3e4c359e715ef63747c40d (DEFAULT_TABLE table=default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6 [id=150e96b3fa8f4fbd8dd20d8413f3483a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.349591 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17142ffce3e4c359e715ef63747c40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.350101 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac (DEFAULT_TABLE table=default.Table_7c57b7890d1f4c8e8e9801e845e4df11 [id=ceebd69263174829ae64d1d71d7d1f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.350499 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.352092 1178 tablet_bootstrap.cc:654] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.354777 896 tablet_service.cc:1505] Processing CreateTablet for tablet 23a78a86cac3421695ac597cf88a5d6e (DEFAULT_TABLE table=default.Table_a3002c1ea00f4683adab3b90c76dde56 [id=eca1a020d00848aeb636f4bfe120f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.355232 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a78a86cac3421695ac597cf88a5d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.359333 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 7492195a41664ef69554d8a6cb148e21 (DEFAULT_TABLE table=default.Table_4b68078657014c3790ceed68233e826e [id=15cdc3d5561948b991ed50cefeded756]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.359465 1044 tablet_service.cc:1505] Processing CreateTablet for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a (DEFAULT_TABLE table=default.Table_d56c826be1cc4fc1a9a8c42bf71799b1 [id=f4b6717002df46beb4bd34daef2d01ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.359737 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7492195a41664ef69554d8a6cb148e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.359920 1036 tablet_service.cc:1505] Processing CreateTablet for tablet e8346fa8652c4233a289b539875e0507 (DEFAULT_TABLE table=default.Table_f29176d94379462f980b648a202d8a6a [id=c5d9ae66f1ed4012ad743406a5e42f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.360378 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8346fa8652c4233a289b539875e0507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.284267 1177 tablet_bootstrap.cc:492] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.360957 1177 ts_tablet_manager.cc:1403] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.129s user 0.015s sys 0.001s
I20251029 08:52:20.363448 1176 tablet_replica.cc:333] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.363528 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.363667 1176 raft_consensus.cc:2243] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.363818 1176 raft_consensus.cc:2272] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.364285 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a17142ffce3e4c359e715ef63747c40d (DEFAULT_TABLE table=default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6 [id=150e96b3fa8f4fbd8dd20d8413f3483a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.364349 1176 ts_tablet_manager.cc:1916] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.364598 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17142ffce3e4c359e715ef63747c40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.364657 1177 raft_consensus.cc:359] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.364837 1177 raft_consensus.cc:385] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.364881 1177 raft_consensus.cc:740] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.365047 1177 consensus_queue.cc:260] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.367760 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 1dd62833c9d548adabbbf71856a4b181 (table default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]) successfully deleted
I20251029 08:52:20.359344 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 18002650e1934b659611e0fbb74b4601 (DEFAULT_TABLE table=default.Table_bbcfd7574bc64702be1c06a2105814d1 [id=a54133947aac4cecb8ffd33bbe39fa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.368515 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18002650e1934b659611e0fbb74b4601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.370877 1177 ts_tablet_manager.cc:1434] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251029 08:52:20.372381 893 tablet_service.cc:1505] Processing CreateTablet for tablet 250b84904da4492390151c8649f3a742 (DEFAULT_TABLE table=default.Table_2964a005ddb04ec6aae15abc146d4b82 [id=a52fdb875303471184f174596eee3bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.372745 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b84904da4492390151c8649f3a742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.373788 1176 ts_tablet_manager.cc:1929] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.373909 1176 log.cc:1199] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/39fc9612689b46d498d6092bd552da4e
I20251029 08:52:20.374205 1176 ts_tablet_manager.cc:1950] T 39fc9612689b46d498d6092bd552da4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.373446 1177 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:20.375844 1183 tablet_bootstrap.cc:492] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.375993 1183 ts_tablet_manager.cc:1403] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.004s
I20251029 08:52:20.376107 2165 raft_consensus.cc:493] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.376238 2165 raft_consensus.cc:515] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.376833 2165 leader_election.cc:290] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.377583 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c4d57a59684723bfd27c7dda302ef3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.379927 1974 ts_tablet_manager.cc:1929] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.380056 1974 log.cc:1199] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c7024e6eb4124109ad1575d216739fe2
I20251029 08:52:20.380355 1974 ts_tablet_manager.cc:1950] T c7024e6eb4124109ad1575d216739fe2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:20.381047 718 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.380833 1177 tablet_bootstrap.cc:654] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.382476 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c7024e6eb4124109ad1575d216739fe2 (table default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]) successfully deleted
I20251029 08:52:20.388008 1177 tablet_bootstrap.cc:492] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.388299 1177 ts_tablet_manager.cc:1403] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
W20251029 08:52:20.389165 996 leader_election.cc:343] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.389307 996 leader_election.cc:304] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.389497 1177 raft_consensus.cc:359] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.389760 1176 tablet_replica.cc:333] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.389964 1176 raft_consensus.cc:2243] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.389760 1177 raft_consensus.cc:385] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.390106 1176 raft_consensus.cc:2272] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.390194 1177 raft_consensus.cc:740] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.390450 1177 consensus_queue.cc:260] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.390635 1176 ts_tablet_manager.cc:1916] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.392596 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 39fc9612689b46d498d6092bd552da4e (table default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]) successfully deleted
I20251029 08:52:20.394531 758 tablet_service.cc:1505] Processing CreateTablet for tablet 23a78a86cac3421695ac597cf88a5d6e (DEFAULT_TABLE table=default.Table_a3002c1ea00f4683adab3b90c76dde56 [id=eca1a020d00848aeb636f4bfe120f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.394953 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a78a86cac3421695ac597cf88a5d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.395417 1178 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.395534 1178 ts_tablet_manager.cc:1403] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20251029 08:52:20.395732 1177 ts_tablet_manager.cc:1434] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251029 08:52:20.396327 1178 raft_consensus.cc:359] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.396482 1178 raft_consensus.cc:385] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.396526 1178 raft_consensus.cc:740] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.396564 2197 raft_consensus.cc:2749] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.396951 1178 consensus_queue.cc:260] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.397081 1183 raft_consensus.cc:359] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.397251 1183 raft_consensus.cc:385] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.397306 1183 raft_consensus.cc:740] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.397557 1183 consensus_queue.cc:260] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.398265 1183 ts_tablet_manager.cc:1434] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20251029 08:52:20.398839 1183 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.400183 1178 ts_tablet_manager.cc:1434] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:20.400499 1178 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.400925 1183 tablet_bootstrap.cc:654] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.401712 1177 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:20.402566 996 leader_election.cc:343] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:20.402695 996 leader_election.cc:304] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.403448 1177 tablet_bootstrap.cc:654] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.405483 1178 tablet_bootstrap.cc:654] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.406627 1179 ts_tablet_manager.cc:1916] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.406628 896 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e (DEFAULT_TABLE table=default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02 [id=86343735ce5d4274b7f24ab1a9d6355a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.407580 901 tablet_service.cc:1505] Processing CreateTablet for tablet 6ebb55ac3d974e6ea2893b3be0e3337e (DEFAULT_TABLE table=default.Table_6d2760f9ad334421ba99b2e63ae70e0d [id=f439cd0505c24cd2b5beaa8996af429b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.408790 887 tablet_service.cc:1505] Processing CreateTablet for tablet 526cbd8cb2f243dcbc736a5a3b1438a3 (DEFAULT_TABLE table=default.Table_fc7b6408d15e4c2a89eb2454f772395b [id=f9fc0a9ad0b24604a93f3e68de941534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.409893 2197 raft_consensus.cc:2749] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:20.410583 996 leader_election.cc:343] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: e4f80728a7a04583bada485707beb806
I20251029 08:52:20.416096 758 tablet_service.cc:1505] Processing CreateTablet for tablet 250b84904da4492390151c8649f3a742 (DEFAULT_TABLE table=default.Table_2964a005ddb04ec6aae15abc146d4b82 [id=a52fdb875303471184f174596eee3bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.416695 747 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e (DEFAULT_TABLE table=default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02 [id=86343735ce5d4274b7f24ab1a9d6355a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.417143 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3da4b29e3c4208ae2109789f994c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.417717 745 tablet_service.cc:1505] Processing CreateTablet for tablet 526cbd8cb2f243dcbc736a5a3b1438a3 (DEFAULT_TABLE table=default.Table_fc7b6408d15e4c2a89eb2454f772395b [id=f9fc0a9ad0b24604a93f3e68de941534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.417855 996 leader_election.cc:304] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.418018 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526cbd8cb2f243dcbc736a5a3b1438a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.418494 2197 raft_consensus.cc:2749] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:20.419445 996 leader_election.cc:343] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.419566 996 leader_election.cc:304] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.420276 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6ebb55ac3d974e6ea2893b3be0e3337e (DEFAULT_TABLE table=default.Table_6d2760f9ad334421ba99b2e63ae70e0d [id=f439cd0505c24cd2b5beaa8996af429b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.420347 1177 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.420604 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebb55ac3d974e6ea2893b3be0e3337e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.420610 1177 ts_tablet_manager.cc:1403] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20251029 08:52:20.421926 1177 raft_consensus.cc:359] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.422194 1177 raft_consensus.cc:385] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.422271 1177 raft_consensus.cc:740] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.422716 1177 consensus_queue.cc:260] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.426280 1178 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.426438 1178 ts_tablet_manager.cc:1403] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20251029 08:52:20.426915 2188 tablet.cc:1722] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.434665 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b84904da4492390151c8649f3a742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.436729 1183 tablet_bootstrap.cc:492] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.436991 1183 ts_tablet_manager.cc:1403] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20251029 08:52:20.436422 1178 raft_consensus.cc:359] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.437640 1178 raft_consensus.cc:385] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.437979 1178 raft_consensus.cc:740] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.439280 1177 ts_tablet_manager.cc:1434] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251029 08:52:20.439190 1178 consensus_queue.cc:260] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.441038 2165 raft_consensus.cc:493] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.441206 2165 raft_consensus.cc:515] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.441744 2165 leader_election.cc:290] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.442415 2197 raft_consensus.cc:2749] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.442696 2165 raft_consensus.cc:493] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.442734 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.442751 1183 raft_consensus.cc:359] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.442817 2165 raft_consensus.cc:515] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.442910 1183 raft_consensus.cc:385] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.442960 1183 raft_consensus.cc:740] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.443166 1183 consensus_queue.cc:260] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.443365 2165 leader_election.cc:290] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.443804 1183 ts_tablet_manager.cc:1434] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:20.444180 1183 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:20.444164 718 leader_election.cc:343] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.444168 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:20.444643 718 leader_election.cc:343] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.445402 1177 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:20.448092 1178 ts_tablet_manager.cc:1434] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20251029 08:52:20.448930 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5e52e5119f2d4324946d85f98d69f62b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.449698 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e52e5119f2d4324946d85f98d69f62b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.449914 1178 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.456152 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:20.456573 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 930314c8f90f462990f93fab8f7790c2 (table default.Table_11e47ff2a71e439ebfb0e9ba79a95267 [id=5691028224ad4d21b0c0ea667274b8af]). Expected version 1 got 0
I20251029 08:52:20.457335 1183 tablet_bootstrap.cc:654] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.457340 1178 tablet_bootstrap.cc:654] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.459916 745 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.460316 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.460773 745 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.461205 745 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.462719 1176 ts_tablet_manager.cc:1929] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.462882 1176 log.cc:1199] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7b5051ff776e4975b917db0393a1f945
I20251029 08:52:20.463279 1176 ts_tablet_manager.cc:1950] T 7b5051ff776e4975b917db0393a1f945 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.464416 1176 tablet_replica.cc:333] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.464651 1176 raft_consensus.cc:2243] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.464854 1176 raft_consensus.cc:2272] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.465426 1176 ts_tablet_manager.cc:1916] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.471861 1179 ts_tablet_manager.cc:1929] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.472114 1179 log.cc:1199] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c7024e6eb4124109ad1575d216739fe2
I20251029 08:52:20.472361 2144 tablet.cc:1722] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.472523 1179 ts_tablet_manager.cc:1950] T c7024e6eb4124109ad1575d216739fe2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.473363 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 757) took 1656 ms. Trace:
I20251029 08:52:20.473527 1179 rpcz_store.cc:276] 1029 08:52:18.816483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.816842 (+ 359us) service_pool.cc:225] Handling call
1029 08:52:20.222399 (+1405557us) ts_tablet_manager.cc:1151] Deleting tablet c7024e6eb4124109ad1575d216739fe2
1029 08:52:20.222410 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.406540 (+184130us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x611001cdf830. stack: 00007f0aa8bae9f6 00007f0aa8bae7b0 00007f0aa7b8a3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f0abc77afbd 00007f0ab5c983be 00007f0ab5dc6f47 00007f0ab8987287 00007f0ab89ba2d4 00007f0ab899ad70 00007f0ab899aafd 0000000000a8867c 00007f0aa8bf4d7f 00007f0aa8bf804b
1029 08:52:20.410424 (+ 3884us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.447499 (+ 37075us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.473343 (+ 25844us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10001,"spinlock_wait_cycles":9396608,"tablet-delete.queue_time_us":1405367}
I20251029 08:52:20.474171 1179 tablet_replica.cc:333] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.474447 1179 raft_consensus.cc:2243] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.474695 1179 raft_consensus.cc:2272] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.474841 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0d00a6b772d4034ac7fa764884c7081 consensus state has no leader
I20251029 08:52:20.475338 1179 ts_tablet_manager.cc:1916] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.444940 2187 tablet.cc:1722] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:20.476820 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c4d57a59684723bfd27c7dda302ef3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.478869 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.479554 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.480155 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3da4b29e3c4208ae2109789f994c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.480350 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526cbd8cb2f243dcbc736a5a3b1438a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.484292 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebb55ac3d974e6ea2893b3be0e3337e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.485648 1183 tablet_bootstrap.cc:492] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.485857 1183 ts_tablet_manager.cc:1403] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20251029 08:52:20.486748 1183 raft_consensus.cc:359] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.487061 1183 raft_consensus.cc:385] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.487167 1183 raft_consensus.cc:740] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.487337 2165 raft_consensus.cc:493] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.487475 2165 raft_consensus.cc:515] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.487498 1183 consensus_queue.cc:260] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.488137 2165 leader_election.cc:290] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.488834 1183 ts_tablet_manager.cc:1434] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:20.489585 1183 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.490007 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa411391c63146e7acb8b4c6edbe12a0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:20.490543 718 leader_election.cc:343] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.491421 1183 tablet_bootstrap.cc:654] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:20.497814 720 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.497957 720 leader_election.cc:304] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:20.498384 720 leader_election.cc:343] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.498472 720 leader_election.cc:304] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.498754 2038 raft_consensus.cc:2749] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.498883 2165 raft_consensus.cc:2749] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.495997 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.496255 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa411391c63146e7acb8b4c6edbe12a0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.496912 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 250b84904da4492390151c8649f3a742 (DEFAULT_TABLE table=default.Table_2964a005ddb04ec6aae15abc146d4b82 [id=a52fdb875303471184f174596eee3bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.500213 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b84904da4492390151c8649f3a742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.500412 1178 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.500598 1178 ts_tablet_manager.cc:1403] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.003s
I20251029 08:52:20.501405 1178 raft_consensus.cc:359] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.501641 1178 raft_consensus.cc:385] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.501749 1178 raft_consensus.cc:740] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.502027 1178 consensus_queue.cc:260] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.502372 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c7024e6eb4124109ad1575d216739fe2 (table default.Table_cc01b9de1eba414ba802605ef8240b11 [id=7810c5e731b3486a83e3da1072f9c0a3]) successfully deleted
I20251029 08:52:20.502657 1178 ts_tablet_manager.cc:1434] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.502789 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7b5051ff776e4975b917db0393a1f945 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:20.503084 1178 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.503197 2196 raft_consensus.cc:493] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.503320 2196 raft_consensus.cc:515] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.503912 2196 leader_election.cc:290] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:20.497083 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 526cbd8cb2f243dcbc736a5a3b1438a3 (DEFAULT_TABLE table=default.Table_fc7b6408d15e4c2a89eb2454f772395b [id=f9fc0a9ad0b24604a93f3e68de941534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.497262 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e (DEFAULT_TABLE table=default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02 [id=86343735ce5d4274b7f24ab1a9d6355a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.504362 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526cbd8cb2f243dcbc736a5a3b1438a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.504599 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3da4b29e3c4208ae2109789f994c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.505116 1177 tablet_bootstrap.cc:654] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.505791 1179 ts_tablet_manager.cc:1929] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.505883 1179 log.cc:1199] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/07906300f1d54d4e8fde5165ee14c44f
I20251029 08:52:20.506141 1179 ts_tablet_manager.cc:1950] T 07906300f1d54d4e8fde5165ee14c44f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.506901 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 756) took 1690 ms. Trace:
I20251029 08:52:20.506980 1179 rpcz_store.cc:276] 1029 08:52:18.816405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.837087 (+ 20682us) service_pool.cc:225] Handling call
1029 08:52:20.474105 (+1637018us) ts_tablet_manager.cc:1151] Deleting tablet 07906300f1d54d4e8fde5165ee14c44f
1029 08:52:20.474121 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.494095 (+ 19974us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.505703 (+ 11608us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.506875 (+ 1172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636738}
I20251029 08:52:20.507275 1179 tablet_replica.cc:333] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.507448 1179 raft_consensus.cc:2243] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.507602 1179 raft_consensus.cc:2272] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.497553 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 6ebb55ac3d974e6ea2893b3be0e3337e (DEFAULT_TABLE table=default.Table_6d2760f9ad334421ba99b2e63ae70e0d [id=f439cd0505c24cd2b5beaa8996af429b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.508586 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebb55ac3d974e6ea2893b3be0e3337e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.511442 2066 server_base.cc:1047] running on GCE node
W20251029 08:52:20.511415 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 08:52:20.512588 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:20.513341 1183 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.513486 1183 ts_tablet_manager.cc:1403] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.002s
W20251029 08:52:20.513857 861 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:20.514323 1183 raft_consensus.cc:359] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.514549 1183 raft_consensus.cc:385] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.514642 1183 raft_consensus.cc:740] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.515280 1183 consensus_queue.cc:260] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.515998 1183 ts_tablet_manager.cc:1434] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 08:52:20.516319 1179 ts_tablet_manager.cc:1916] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.504360 1176 ts_tablet_manager.cc:1929] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.521090 1176 log.cc:1199] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:20.521116 1177 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.521263 1177 ts_tablet_manager.cc:1403] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.001s
I20251029 08:52:20.521561 1176 ts_tablet_manager.cc:1950] T a2b7853b85d24a04b8b11caaa4fe8c4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.522017 1177 raft_consensus.cc:359] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.522202 1177 raft_consensus.cc:385] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.522500 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14123bfd9c3a4a6aba8cacd92e0ac0bc" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.522797 1176 tablet_replica.cc:333] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.522960 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a2b7853b85d24a04b8b11caaa4fe8c4d (table default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]) successfully deleted
I20251029 08:52:20.523028 1176 raft_consensus.cc:2243] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:20.523044 861 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.523157 1176 raft_consensus.cc:2272] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.523157 861 leader_election.cc:304] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:20.522257 1177 raft_consensus.cc:740] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.523561 2196 raft_consensus.cc:2749] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.524096 2197 raft_consensus.cc:493] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.524227 2197 raft_consensus.cc:515] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.524158 1176 ts_tablet_manager.cc:1916] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.524786 2197 leader_election.cc:290] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:20.525039 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 08:52:20.525318 1183 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:20.525686 998 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:20.526038 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66c918ae6920491b9e2fc399447c8bb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.526418 2066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20251029 08:52:20.526719 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66c918ae6920491b9e2fc399447c8bb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.529088 1183 tablet_bootstrap.cc:654] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.529701 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7671e29e48fc478c900959b22a0b98b3
I20251029 08:52:20.529872 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 485c9946f3f14a2aab11f4cdaa70a5b6
I20251029 08:52:20.531654 1176 ts_tablet_manager.cc:1929] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.531761 1176 log.cc:1199] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:20.531771 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4e213713757949caa3f504f70026b7b5
I20251029 08:52:20.533612 1178 tablet_bootstrap.cc:654] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.533965 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:20.534415 597 catalog_manager.cc:6719] ReplaceTablet: tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 deleted and replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65
I20251029 08:52:20.534389 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.535008 1179 ts_tablet_manager.cc:1929] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.535101 1179 log.cc:1199] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bc9c221eaa9a4721ba25761206a13645
I20251029 08:52:20.535394 1179 ts_tablet_manager.cc:1950] T bc9c221eaa9a4721ba25761206a13645 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.535542 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.535779 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 65b20d53d6214bdc81ca205dbf922d7b
W20251029 08:52:20.536044 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4e213713757949caa3f504f70026b7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e213713757949caa3f504f70026b7b5 already in progress: creating tablet
I20251029 08:52:20.537127 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:20.537707 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 already in progress: creating tablet
W20251029 08:52:20.538177 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 65b20d53d6214bdc81ca205dbf922d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65b20d53d6214bdc81ca205dbf922d7b already in progress: creating tablet
I20251029 08:52:20.539505 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 7671e29e48fc478c900959b22a0b98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.539819 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.540153 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.540664 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.540773 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.540807 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.541285 1183 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.541502 1183 ts_tablet_manager.cc:1403] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
W20251029 08:52:20.541416 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485c9946f3f14a2aab11f4cdaa70a5b6 already in progress: creating tablet
I20251029 08:52:20.541852 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 7671e29e48fc478c900959b22a0b98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.542341 1183 raft_consensus.cc:359] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.542608 1183 raft_consensus.cc:385] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:20.542659 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7671e29e48fc478c900959b22a0b98b3 because the tablet was not found. No further retry: Not found: Tablet not found: 7671e29e48fc478c900959b22a0b98b3
I20251029 08:52:20.542713 1183 raft_consensus.cc:740] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.543406 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.543939 1183 consensus_queue.cc:260] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.545076 1183 ts_tablet_manager.cc:1434] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:20.545431 1183 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.547142 1183 tablet_bootstrap.cc:654] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.548314 1177 consensus_queue.cc:260] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.548903 1178 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.549052 1178 ts_tablet_manager.cc:1403] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.001s
I20251029 08:52:20.549893 1178 raft_consensus.cc:359] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.550064 1178 raft_consensus.cc:385] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.550120 1178 raft_consensus.cc:740] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.550335 1178 consensus_queue.cc:260] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.551601 1178 ts_tablet_manager.cc:1434] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.551494 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 762) took 1555 ms. Trace:
I20251029 08:52:20.551652 896 tablet_service.cc:1505] Processing CreateTablet for tablet fc32b18886d74e3191dadad2b0bfbdb7 (DEFAULT_TABLE table=default.Table_0921613fbbaa49ae95be6d937ab40033 [id=2dfc22bc74d744acae551e2a8f31a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.552141 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc32b18886d74e3191dadad2b0bfbdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.552675 1178 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.552855 1179 rpcz_store.cc:276] 1029 08:52:18.996434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:18.996483 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:20.507231 (+1510748us) ts_tablet_manager.cc:1151] Deleting tablet bc9c221eaa9a4721ba25761206a13645
1029 08:52:20.507240 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.531740 (+ 24500us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.534917 (+ 3177us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.551449 (+ 16532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510441}
I20251029 08:52:20.553293 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 766) took 1451 ms. Trace:
I20251029 08:52:20.553449 1179 rpcz_store.cc:276] 1029 08:52:19.101872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.102046 (+ 174us) service_pool.cc:225] Handling call
1029 08:52:20.553142 (+1451096us) ts_tablet_manager.cc:1151] Deleting tablet bda3e143699b41b6bba308c83bbb3e0d
1029 08:52:20.553152 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.553261 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450918}
I20251029 08:52:20.553573 892 tablet_service.cc:1505] Processing CreateTablet for tablet d8d23a562f954df7afdb999af8cb8dce (DEFAULT_TABLE table=default.Table_684cbdb8f5aa41cca591cf8b4e991c11 [id=18e2dee6642542fe976c4f05ccc6e661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.553746 1179 tablet_replica.cc:333] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.553952 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d23a562f954df7afdb999af8cb8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.553947 1179 raft_consensus.cc:2243] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.554111 1179 raft_consensus.cc:2272] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.554379 1178 tablet_bootstrap.cc:654] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.554721 1179 ts_tablet_manager.cc:1916] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.554880 902 tablet_service.cc:1505] Processing CreateTablet for tablet ae11b4ff92d54e3e8a97a94e950ca078 (DEFAULT_TABLE table=default.Table_f0f2d3061ba74387a865d0f5f47598f2 [id=0470dc08e273430ca8f0d70dbf983795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.555234 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae11b4ff92d54e3e8a97a94e950ca078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.555258 1176 ts_tablet_manager.cc:1950] T a8c4d57a59684723bfd27c7dda302ef3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.555848 888 tablet_service.cc:1505] Processing CreateTablet for tablet 647c637b00044033a1f6d461a67b6500 (DEFAULT_TABLE table=default.Table_cbf544afceae424696ddc50a2292d48b [id=1d632005098049aa89f3ca66616a4f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.556249 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c637b00044033a1f6d461a67b6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.556476 1176 tablet_replica.cc:333] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.556648 1176 raft_consensus.cc:2243] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.556795 1176 raft_consensus.cc:2272] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.557047 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 23a78a86cac3421695ac597cf88a5d6e (DEFAULT_TABLE table=default.Table_a3002c1ea00f4683adab3b90c76dde56 [id=eca1a020d00848aeb636f4bfe120f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.557499 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a78a86cac3421695ac597cf88a5d6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:20.558296 996 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 60585.1us
I20251029 08:52:20.558871 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c510c2d3e9184bcb92acb666b6bbfb84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.559119 720 leader_election.cc:343] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.559233 720 leader_election.cc:304] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:20.559512 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c510c2d3e9184bcb92acb666b6bbfb84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.559643 2165 raft_consensus.cc:2749] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:20.560137 720 leader_election.cc:343] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.560249 720 leader_election.cc:304] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.560581 2165 raft_consensus.cc:2749] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.560880 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f1dca37ca0fc4bc38418e729d7fbff0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.561255 1176 ts_tablet_manager.cc:1916] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:20.561386 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1dca37ca0fc4bc38418e729d7fbff0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.562374 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.562613 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 7671e29e48fc478c900959b22a0b98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.562813 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:20.562875 996 leader_election.cc:343] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:20.563015 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.563138 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet a20f26d0f7874ccb960ffe4d27e854e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.563270 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.564673 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:20.564626 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_919fdbb0653a4eb39f8cec21d90854c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.565500 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_919fdbb0653a4eb39f8cec21d90854c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.564997 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_580fb63e5fcb4335990145e4f4fe128c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.566984 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_580fb63e5fcb4335990145e4f4fe128c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.570003 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_020f3afc90994322af7afdfce26b6e7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.570655 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:20.572626 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7f838f98d4a34099a018852bfa404b97" } new_table_name: "default.Table_70588860077547cabad70977368b56fc" modify_external_catalogs: true
I20251029 08:52:20.573091 1178 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.573223 1178 ts_tablet_manager.cc:1403] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20251029 08:52:20.574023 1178 raft_consensus.cc:359] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.574185 1178 raft_consensus.cc:385] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.574231 1178 raft_consensus.cc:740] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.574460 1178 consensus_queue.cc:260] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.574549 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.574661 891 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.574832 1974 tablet_replica.cc:333] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:20.575011 1974 raft_consensus.cc:2243] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.575088 1178 ts_tablet_manager.cc:1434] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.575178 1974 raft_consensus.cc:2272] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.575380 1178 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:20.575518 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_020f3afc90994322af7afdfce26b6e7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.575937 901 tablet_service.cc:1505] Processing CreateTablet for tablet e14d1d3421a542ec960d21453bb713ec (DEFAULT_TABLE table=default.Table_836882de6c804a5c908824f31f0aa328 [id=2eb80c275e8945e4a0e34f90f5d2d763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.576273 1974 ts_tablet_manager.cc:1916] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.576328 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d1d3421a542ec960d21453bb713ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.578883 1178 tablet_bootstrap.cc:654] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.579960 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 07906300f1d54d4e8fde5165ee14c44f (table default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4 [id=1c8816c838dd4770a4e47258459e9c0c]) successfully deleted
I20251029 08:52:20.580878 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bc9c221eaa9a4721ba25761206a13645 (table default.Table_1e141ef36f5541e5bba246df7272d2ec [id=348455c16bce4888b3b32e057c2eec08]) successfully deleted
W20251029 08:52:20.581230 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bda3e143699b41b6bba308c83bbb3e0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bda3e143699b41b6bba308c83bbb3e0d already in progress: creating tablet
I20251029 08:52:20.581480 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a8c4d57a59684723bfd27c7dda302ef3 (table default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]) successfully deleted
I20251029 08:52:20.579593 748 tablet_service.cc:1505] Processing CreateTablet for tablet e14d1d3421a542ec960d21453bb713ec (DEFAULT_TABLE table=default.Table_836882de6c804a5c908824f31f0aa328 [id=2eb80c275e8945e4a0e34f90f5d2d763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.582305 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d1d3421a542ec960d21453bb713ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.579704 745 tablet_service.cc:1505] Processing CreateTablet for tablet d8d23a562f954df7afdb999af8cb8dce (DEFAULT_TABLE table=default.Table_684cbdb8f5aa41cca591cf8b4e991c11 [id=18e2dee6642542fe976c4f05ccc6e661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.584391 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d23a562f954df7afdb999af8cb8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.585747 1183 tablet_bootstrap.cc:492] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.585904 1183 ts_tablet_manager.cc:1403] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20251029 08:52:20.586691 1183 raft_consensus.cc:359] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.586862 1183 raft_consensus.cc:385] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.586934 1183 raft_consensus.cc:740] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.587230 1183 consensus_queue.cc:260] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.587872 1183 ts_tablet_manager.cc:1434] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.588196 1183 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.589789 1183 tablet_bootstrap.cc:654] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.563033 996 leader_election.cc:304] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.592365 2044 raft_consensus.cc:2749] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.592407 1176 ts_tablet_manager.cc:1929] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.592502 1176 log.cc:1199] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:20.592818 1176 ts_tablet_manager.cc:1950] T e8c4d793f10147fba5e4f44474575bf0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.594331 1179 ts_tablet_manager.cc:1929] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.594484 1179 log.cc:1199] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e626994e96d54c0d913fdedc85fd1155
I20251029 08:52:20.594766 1179 ts_tablet_manager.cc:1950] T e626994e96d54c0d913fdedc85fd1155 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.580003 758 tablet_service.cc:1505] Processing CreateTablet for tablet fc32b18886d74e3191dadad2b0bfbdb7 (DEFAULT_TABLE table=default.Table_0921613fbbaa49ae95be6d937ab40033 [id=2dfc22bc74d744acae551e2a8f31a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.580233 747 tablet_service.cc:1505] Processing CreateTablet for tablet ae11b4ff92d54e3e8a97a94e950ca078 (DEFAULT_TABLE table=default.Table_f0f2d3061ba74387a865d0f5f47598f2 [id=0470dc08e273430ca8f0d70dbf983795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.595606 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 765) took 1493 ms. Trace:
I20251029 08:52:20.595701 1179 rpcz_store.cc:276] 1029 08:52:19.101760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.134450 (+ 32690us) service_pool.cc:225] Handling call
1029 08:52:20.553680 (+1419230us) ts_tablet_manager.cc:1151] Deleting tablet e626994e96d54c0d913fdedc85fd1155
1029 08:52:20.553692 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.581709 (+ 28017us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.594218 (+ 12509us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.595545 (+ 1327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419003}
I20251029 08:52:20.596182 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 770) took 1381 ms. Trace:
I20251029 08:52:20.596271 1179 rpcz_store.cc:276] 1029 08:52:19.214339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.219128 (+ 4789us) service_pool.cc:225] Handling call
1029 08:52:20.595989 (+1376861us) ts_tablet_manager.cc:1151] Deleting tablet cf09e8fd3d2b43318fa37d922c8b826f
1029 08:52:20.596000 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.596132 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376595}
I20251029 08:52:20.596633 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 771) took 1379 ms. Trace:
I20251029 08:52:20.596725 1179 rpcz_store.cc:276] 1029 08:52:19.216864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.222522 (+ 5658us) service_pool.cc:225] Handling call
1029 08:52:20.596500 (+1373978us) ts_tablet_manager.cc:1151] Deleting tablet 62b7fe26c7ad48cdb935e8dc6b1962a7
1029 08:52:20.596508 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.596576 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373808}
I20251029 08:52:20.596987 1179 tablet_replica.cc:333] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.597198 1179 raft_consensus.cc:2243] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.597422 1179 raft_consensus.cc:2272] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.597960 1179 ts_tablet_manager.cc:1916] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.599588 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e626994e96d54c0d913fdedc85fd1155 (table default.Table_742ee97669df417aba70fdcea4983c94 [id=185a9dcf828b43dcb47d8375313c0f2c]) successfully deleted
I20251029 08:52:20.600638 1043 tablet_service.cc:1505] Processing CreateTablet for tablet e14d1d3421a542ec960d21453bb713ec (DEFAULT_TABLE table=default.Table_836882de6c804a5c908824f31f0aa328 [id=2eb80c275e8945e4a0e34f90f5d2d763]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:20.600879 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cf09e8fd3d2b43318fa37d922c8b826f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf09e8fd3d2b43318fa37d922c8b826f already in progress: creating tablet
I20251029 08:52:20.601056 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14d1d3421a542ec960d21453bb713ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.601457 1036 tablet_service.cc:1505] Processing CreateTablet for tablet d8d23a562f954df7afdb999af8cb8dce (DEFAULT_TABLE table=default.Table_684cbdb8f5aa41cca591cf8b4e991c11 [id=18e2dee6642542fe976c4f05ccc6e661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.601825 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d23a562f954df7afdb999af8cb8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.604305 1035 tablet_service.cc:1505] Processing CreateTablet for tablet ae11b4ff92d54e3e8a97a94e950ca078 (DEFAULT_TABLE table=default.Table_f0f2d3061ba74387a865d0f5f47598f2 [id=0470dc08e273430ca8f0d70dbf983795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.604424 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet e4f80728a7a04583bada485707beb806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.604707 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae11b4ff92d54e3e8a97a94e950ca078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.609877 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet fd25eeb89e574297a92d8e978a79d768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.604298 1044 tablet_service.cc:1505] Processing CreateTablet for tablet fc32b18886d74e3191dadad2b0bfbdb7 (DEFAULT_TABLE table=default.Table_0921613fbbaa49ae95be6d937ab40033 [id=2dfc22bc74d744acae551e2a8f31a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.610939 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc32b18886d74e3191dadad2b0bfbdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.612598 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae11b4ff92d54e3e8a97a94e950ca078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.612655 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet d30183814f5e4f2ab9d71fe493eb801c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:07 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.614019 757 tablet_service.cc:1505] Processing CreateTablet for tablet 647c637b00044033a1f6d461a67b6500 (DEFAULT_TABLE table=default.Table_cbf544afceae424696ddc50a2292d48b [id=1d632005098049aa89f3ca66616a4f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.614403 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c637b00044033a1f6d461a67b6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.614562 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a712185fe4aa4f8faf1ce2df229c11ee
I20251029 08:52:20.617899 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.617959 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.618070 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 840ff254244f43a18679232279e02d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.618889 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 8991ea78321545be85d5c480841e4751 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ad04f57c29407b9d2420520c095a28) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.620321 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc32b18886d74e3191dadad2b0bfbdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.621366 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 647c637b00044033a1f6d461a67b6500 (DEFAULT_TABLE table=default.Table_cbf544afceae424696ddc50a2292d48b [id=1d632005098049aa89f3ca66616a4f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.621780 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c637b00044033a1f6d461a67b6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.625830 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]). Expected version 2 got 0
I20251029 08:52:20.627439 1176 tablet_replica.cc:333] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.627648 1176 raft_consensus.cc:2243] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.627772 1176 raft_consensus.cc:2272] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.628369 1176 ts_tablet_manager.cc:1916] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.628931 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e8c4d793f10147fba5e4f44474575bf0 (table default.Table_02acde1d2bfe41e9a0f7565ed487c1c7 [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:20.629252 888 tablet_service.cc:1505] Processing CreateTablet for tablet 1636c78f17a549d7b97dfacee192f8a4 (DEFAULT_TABLE table=default.Table_2067fe9013134f5b945ace2150cd86d2 [id=087101bcf93f40ad9ac9b154ba7ae766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.630033 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:20.630332 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62b7fe26c7ad48cdb935e8dc6b1962a7 already in progress: creating tablet
I20251029 08:52:20.632356 902 tablet_service.cc:1505] Processing CreateTablet for tablet 436dd46ff4a04a9bbc957603880af7c0 (DEFAULT_TABLE table=default.Table_d18c461ffec64795a25d91d924ffa054 [id=fca46ccf98cf40b58066c79b9a10e435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.632762 748 tablet_service.cc:1505] Processing CreateTablet for tablet 1636c78f17a549d7b97dfacee192f8a4 (DEFAULT_TABLE table=default.Table_2067fe9013134f5b945ace2150cd86d2 [id=087101bcf93f40ad9ac9b154ba7ae766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.633111 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1636c78f17a549d7b97dfacee192f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.633706 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 1636c78f17a549d7b97dfacee192f8a4 (DEFAULT_TABLE table=default.Table_2067fe9013134f5b945ace2150cd86d2 [id=087101bcf93f40ad9ac9b154ba7ae766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.634071 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1636c78f17a549d7b97dfacee192f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.638319 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 436dd46ff4a04a9bbc957603880af7c0 (DEFAULT_TABLE table=default.Table_d18c461ffec64795a25d91d924ffa054 [id=fca46ccf98cf40b58066c79b9a10e435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.638736 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436dd46ff4a04a9bbc957603880af7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.639693 2044 raft_consensus.cc:493] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.639819 2044 raft_consensus.cc:515] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.640431 2044 leader_election.cc:290] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.641119 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 31bed64a27164baa8177b87b54b5947b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:20.641453 590 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde] alter complete (version 2)
W20251029 08:52:20.641539 998 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:20.651190 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436dd46ff4a04a9bbc957603880af7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.651302 1178 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.651360 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ffc678b874871a672c3059235de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.651463 1178 ts_tablet_manager.cc:1403] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.004s
I20251029 08:52:20.652230 1974 ts_tablet_manager.cc:1929] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.652348 1974 log.cc:1199] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:20.652488 1178 raft_consensus.cc:359] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.652683 1974 ts_tablet_manager.cc:1950] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:20.652730 1178 raft_consensus.cc:385] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.652782 1178 raft_consensus.cc:740] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.652956 1178 consensus_queue.cc:260] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.653257 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (table default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]) successfully deleted
I20251029 08:52:20.653656 1178 ts_tablet_manager.cc:1434] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.653861 1974 tablet_replica.cc:333] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:20.654052 1974 raft_consensus.cc:2243] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.654182 1178 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.654225 1974 raft_consensus.cc:2272] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.654421 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.661581 1178 tablet_bootstrap.cc:654] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.661979 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_45ce89ee1267486fb1dc5d6990cb0716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.662549 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:20.662676 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45ce89ee1267486fb1dc5d6990cb0716 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.667039 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.668970 1176 ts_tablet_manager.cc:1929] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.669092 1176 log.cc:1199] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bd0c43367a8c47f390cc61d2764d61e4
I20251029 08:52:20.669411 1176 ts_tablet_manager.cc:1950] T bd0c43367a8c47f390cc61d2764d61e4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.669723 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:20.669757 996 leader_election.cc:343] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:20.669881 996 leader_election.cc:304] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.670118 2044 raft_consensus.cc:2749] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.671036 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bd0c43367a8c47f390cc61d2764d61e4 (table default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]) successfully deleted
I20251029 08:52:20.671805 1974 ts_tablet_manager.cc:1916] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.674566 1179 ts_tablet_manager.cc:1929] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:20.674678 1179 log.cc:1199] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c4e1b6b6359b437aa295b473fa42dce1
I20251029 08:52:20.675051 1179 ts_tablet_manager.cc:1950] T c4e1b6b6359b437aa295b473fa42dce1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.675798 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 772) took 1455 ms. Trace:
I20251029 08:52:20.675891 1179 rpcz_store.cc:276] 1029 08:52:19.219877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.220483 (+ 606us) service_pool.cc:225] Handling call
1029 08:52:20.596911 (+1376428us) ts_tablet_manager.cc:1151] Deleting tablet c4e1b6b6359b437aa295b473fa42dce1
1029 08:52:20.596921 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.633288 (+ 36367us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.674453 (+ 41165us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.675778 (+ 1325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371806}
I20251029 08:52:20.676345 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 773) took 1421 ms. Trace:
I20251029 08:52:20.550570 1177 ts_tablet_manager.cc:1434] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251029 08:52:20.676446 1179 rpcz_store.cc:276] 1029 08:52:19.254822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.254882 (+ 60us) service_pool.cc:225] Handling call
1029 08:52:20.676211 (+1421329us) ts_tablet_manager.cc:1151] Deleting tablet e14d05e64689423da77d6d60b760cc06
1029 08:52:20.676223 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.676329 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1421041}
I20251029 08:52:20.676771 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c7024e6eb4124109ad1575d216739fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 90360621ce124475af2f12060471f63c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.676911 1178 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.677049 1178 ts_tablet_manager.cc:1403] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20251029 08:52:20.677058 887 tablet_service.cc:1505] Processing CreateTablet for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d (DEFAULT_TABLE table=default.Table_b843daec43c24b55a07bbfa6d6fc5a5f [id=76822b2441764f5294d0a7155cd350ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.677798 1178 raft_consensus.cc:359] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.677971 1178 raft_consensus.cc:385] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.678036 1178 raft_consensus.cc:740] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.678056 1177 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:20.678278 1178 consensus_queue.cc:260] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.678855 1178 ts_tablet_manager.cc:1434] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.679102 1178 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.679683 1177 tablet_bootstrap.cc:654] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.680927 1178 tablet_bootstrap.cc:654] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.681756 1176 tablet_replica.cc:333] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.681934 1176 raft_consensus.cc:2243] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.682092 1176 raft_consensus.cc:2272] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.682840 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.682917 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 774) took 1409 ms. Trace:
I20251029 08:52:20.683012 1179 rpcz_store.cc:276] 1029 08:52:19.273841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.374477 (+100636us) service_pool.cc:225] Handling call
1029 08:52:20.682769 (+1308292us) ts_tablet_manager.cc:1151] Deleting tablet 663d6587a5e14519afbb0f94d691d1b4
1029 08:52:20.682782 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.682894 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219793}
I20251029 08:52:20.683380 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 781) took 1295 ms. Trace:
I20251029 08:52:20.683461 1179 rpcz_store.cc:276] 1029 08:52:19.387951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.388057 (+ 106us) service_pool.cc:225] Handling call
1029 08:52:20.683280 (+1295223us) ts_tablet_manager.cc:1151] Deleting tablet 47f58be6d7f5492395ca6192f7ce0930
1029 08:52:20.683289 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.683367 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211170}
I20251029 08:52:20.683674 1179 tablet_replica.cc:333] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.683864 1179 raft_consensus.cc:2243] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.684006 1179 raft_consensus.cc:2272] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.684517 1179 ts_tablet_manager.cc:1916] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.684930 1176 ts_tablet_manager.cc:1916] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.693017 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c4e1b6b6359b437aa295b473fa42dce1 (table default.Table_c0e519f0c5604145a2472a038ef78e46 [id=260ff4d973084d2b9572c2f4703c9734]) successfully deleted
W20251029 08:52:20.693470 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e14d05e64689423da77d6d60b760cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d05e64689423da77d6d60b760cc06 already in progress: creating tablet
W20251029 08:52:20.693805 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 663d6587a5e14519afbb0f94d691d1b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663d6587a5e14519afbb0f94d691d1b4 already in progress: creating tablet
W20251029 08:52:20.694114 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 47f58be6d7f5492395ca6192f7ce0930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47f58be6d7f5492395ca6192f7ce0930 already in progress: creating tablet
I20251029 08:52:20.694653 758 tablet_service.cc:1505] Processing CreateTablet for tablet 436dd46ff4a04a9bbc957603880af7c0 (DEFAULT_TABLE table=default.Table_d18c461ffec64795a25d91d924ffa054 [id=fca46ccf98cf40b58066c79b9a10e435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.695513 1178 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.695647 1178 ts_tablet_manager.cc:1403] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20251029 08:52:20.696578 1178 raft_consensus.cc:359] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.696751 1178 raft_consensus.cc:385] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.696803 1178 raft_consensus.cc:740] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.697028 1178 consensus_queue.cc:260] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.697678 1178 ts_tablet_manager.cc:1434] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.698283 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5f659bad54be4c2390461e5b06833ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:20.698938 1178 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:20.698972 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f659bad54be4c2390461e5b06833ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.700177 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c6c5865ac9e944ea97b59f1b9e274284" } new_table_name: "default.Table_c9aecdc6faf64b93ad71b9a5bcbd6199" modify_external_catalogs: true
I20251029 08:52:20.701064 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436dd46ff4a04a9bbc957603880af7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.701956 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c15ab0db646407ab8460031c72ab852" } new_table_name: "default.Table_72b06540aa1c4b51826638ef3543665e" modify_external_catalogs: true
I20251029 08:52:20.703282 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1c4199c5e69245c5a9e8cae1a4ed98da" } new_table_name: "default.Table_c30d9ab9918b4baaa0a6d733432364c1" modify_external_catalogs: true
I20251029 08:52:20.705953 597 master_service.cc:954] ReplaceTablet: received request to replace tablet fcaa3fa779824b38887ae001623ba67b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:20.706163 1178 tablet_bootstrap.cc:654] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.706354 2165 raft_consensus.cc:493] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.706470 2165 raft_consensus.cc:515] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.707039 2165 leader_election.cc:290] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:20.707460 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be4bcf9e3324bd884e946fae20648fd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.707584 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be4bcf9e3324bd884e946fae20648fd" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:20.708004 720 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:20.708149 718 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.708257 718 leader_election.cc:304] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.708684 2165 raft_consensus.cc:2749] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.709017 2066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 08:52:20.711052 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d (DEFAULT_TABLE table=default.Table_b843daec43c24b55a07bbfa6d6fc5a5f [id=76822b2441764f5294d0a7155cd350ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.711270 748 tablet_service.cc:1505] Processing CreateTablet for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d (DEFAULT_TABLE table=default.Table_b843daec43c24b55a07bbfa6d6fc5a5f [id=76822b2441764f5294d0a7155cd350ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.711410 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ffc678b874871a672c3059235de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.711481 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.711594 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f7ffd87a3d4eb9975a418b3a63fe8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.718940 2044 raft_consensus.cc:493] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.719102 2044 raft_consensus.cc:515] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.719700 2044 leader_election.cc:290] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.720155 2165 raft_consensus.cc:493] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.720288 2165 raft_consensus.cc:515] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.720880 2165 leader_election.cc:290] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.721818 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43c3074caee74069a8a7f1b87721d726" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:20.722481 718 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:20.722760 998 leader_election.cc:343] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:20.723841 1176 ts_tablet_manager.cc:1929] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.723946 1176 log.cc:1199] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:20.723891 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43c3074caee74069a8a7f1b87721d726" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:20.724457 720 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.724566 720 leader_election.cc:304] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.724761 1179 ts_tablet_manager.cc:1929] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.724826 1183 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.724853 1179 log.cc:1199] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/363fcf8a3e0f4770a9743fddffa6495f
I20251029 08:52:20.724961 1183 ts_tablet_manager.cc:1403] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.137s user 0.013s sys 0.010s
I20251029 08:52:20.725131 1179 ts_tablet_manager.cc:1950] T 363fcf8a3e0f4770a9743fddffa6495f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:20.725289 996 leader_election.cc:343] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:20.725502 2165 raft_consensus.cc:2749] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.725735 996 leader_election.cc:304] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.726012 2044 raft_consensus.cc:2749] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.726035 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 788) took 1007 ms. Trace:
I20251029 08:52:20.726111 1179 rpcz_store.cc:276] 1029 08:52:19.718401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.718482 (+ 81us) service_pool.cc:225] Handling call
1029 08:52:20.683637 (+965155us) ts_tablet_manager.cc:1151] Deleting tablet 363fcf8a3e0f4770a9743fddffa6495f
1029 08:52:20.683644 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.701614 (+ 17970us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.724663 (+ 23049us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.726019 (+ 1356us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101428,"tablet-delete.queue_time_us":964835}
I20251029 08:52:20.726429 1179 tablet_replica.cc:333] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.726599 1179 raft_consensus.cc:2243] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.726744 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_34c608523fd44399b9c71a365266edd6" } new_table_name: "default.Table_c0e6b2aa7a434d399ef4384fe8093945" modify_external_catalogs: true
I20251029 08:52:20.725742 1183 raft_consensus.cc:359] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.726890 1183 raft_consensus.cc:385] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.726938 1183 raft_consensus.cc:740] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.727170 1183 consensus_queue.cc:260] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.728318 1176 ts_tablet_manager.cc:1950] T aa411391c63146e7acb8b4c6edbe12a0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.729216 1176 tablet_replica.cc:333] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.729403 1176 raft_consensus.cc:2243] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.729527 1176 raft_consensus.cc:2272] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.730002 1176 ts_tablet_manager.cc:1916] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.726729 1179 raft_consensus.cc:2272] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.731165 1179 ts_tablet_manager.cc:1916] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.731829 1177 tablet_bootstrap.cc:492] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:20.734828 1183 ts_tablet_manager.cc:1434] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 08:52:20.737241 1179 ts_tablet_manager.cc:1929] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.737347 1179 log.cc:1199] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/608b7095d42941ccb0c4eb57f6754e74
I20251029 08:52:20.737696 1179 ts_tablet_manager.cc:1950] T 608b7095d42941ccb0c4eb57f6754e74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.738425 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 787) took 1020 ms. Trace:
I20251029 08:52:20.738516 1179 rpcz_store.cc:276] 1029 08:52:19.718276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:19.718686 (+ 410us) service_pool.cc:225] Handling call
1029 08:52:20.726375 (+1007689us) ts_tablet_manager.cc:1151] Deleting tablet 608b7095d42941ccb0c4eb57f6754e74
1029 08:52:20.726384 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:20.734282 (+ 7898us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.737157 (+ 2875us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:20.738394 (+ 1237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101431,"tablet-delete.queue_time_us":1007515}
I20251029 08:52:20.738938 1178 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.738965 2165 raft_consensus.cc:493] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.739045 1179 tablet_replica.cc:333] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.739082 2165 raft_consensus.cc:515] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.739408 1183 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.739697 2165 leader_election.cc:290] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:20.740152 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4ae5978b71443a48a88bc69128b187d consensus state has no leader
I20251029 08:52:20.739073 1178 ts_tablet_manager.cc:1403] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.001s
I20251029 08:52:20.741238 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba201a54466240129d82a096f9316242" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.741808 1178 raft_consensus.cc:359] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.741956 1178 raft_consensus.cc:385] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:20.741991 720 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.741997 1178 raft_consensus.cc:740] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.742578 1178 consensus_queue.cc:260] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.745115 1179 raft_consensus.cc:2243] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.745299 1179 raft_consensus.cc:2272] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.745740 1178 ts_tablet_manager.cc:1434] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:20.745914 1179 ts_tablet_manager.cc:1916] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.746407 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 363fcf8a3e0f4770a9743fddffa6495f (table default.Table_64f9de5287d54b348a406a42a7c379ab [id=0e01efd35bcd43dc88e2d92f0734b958]) successfully deleted
I20251029 08:52:20.746713 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 608b7095d42941ccb0c4eb57f6754e74 (table default.Table_77fe8da38eac41a6968a6e26fd81d040 [id=eb0a43b4844f4a21a187afa2f54b3664]) successfully deleted
W20251029 08:52:20.747140 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a213a91ac12e46ca9bc15e3bb9d39a97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a213a91ac12e46ca9bc15e3bb9d39a97 already in progress: creating tablet
I20251029 08:52:20.747565 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet aa411391c63146e7acb8b4c6edbe12a0 (table default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]) successfully deleted
I20251029 08:52:20.748442 1178 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.750511 1178 tablet_bootstrap.cc:654] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.750826 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 8553f806c78f4617bb8d97cdba2243bf (DEFAULT_TABLE table=default.Table_5dc644387108496995ffad8f788f3c24 [id=d38e1320dbd54a379e985e5af8840f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.751181 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8553f806c78f4617bb8d97cdba2243bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.751264 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8553f806c78f4617bb8d97cdba2243bf (DEFAULT_TABLE table=default.Table_5dc644387108496995ffad8f788f3c24 [id=d38e1320dbd54a379e985e5af8840f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.740887 1183 tablet_bootstrap.cc:654] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.751580 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8553f806c78f4617bb8d97cdba2243bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.756319 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2b46bb1ed1994c149eb9a1db10b00921 (DEFAULT_TABLE table=default.Table_439c96c50e724a6db3abaac6c5540a36 [id=44c6086367ec4467a96fb0bbb8a749e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.756676 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b46bb1ed1994c149eb9a1db10b00921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.759778 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cc01b9de1eba414ba802605ef8240b11" } modify_external_catalogs: true
I20251029 08:52:20.759965 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cc01b9de1eba414ba802605ef8240b11" } modify_external_catalogs: true
I20251029 08:52:20.760303 757 tablet_service.cc:1505] Processing CreateTablet for tablet 2b46bb1ed1994c149eb9a1db10b00921 (DEFAULT_TABLE table=default.Table_439c96c50e724a6db3abaac6c5540a36 [id=44c6086367ec4467a96fb0bbb8a749e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.760696 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b46bb1ed1994c149eb9a1db10b00921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.764093 2044 raft_consensus.cc:493] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.764237 2044 raft_consensus.cc:515] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.764865 2044 leader_election.cc:290] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:20.765729 998 leader_election.cc:343] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 840ff254244f43a18679232279e02d1f
W20251029 08:52:20.765904 996 leader_election.cc:343] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 840ff254244f43a18679232279e02d1f
I20251029 08:52:20.766003 996 leader_election.cc:304] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.766192 2044 raft_consensus.cc:2749] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.769299 1183 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.769440 1183 ts_tablet_manager.cc:1403] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20251029 08:52:20.770054 1183 raft_consensus.cc:359] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.770217 1183 raft_consensus.cc:385] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.770267 1183 raft_consensus.cc:740] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.770265 1176 ts_tablet_manager.cc:1929] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.770366 1176 log.cc:1199] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/652903cbbdb042d780e7d71fc2c0ab4d
I20251029 08:52:20.770463 1183 consensus_queue.cc:260] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.770648 1176 ts_tablet_manager.cc:1950] T 652903cbbdb042d780e7d71fc2c0ab4d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.772342 1183 ts_tablet_manager.cc:1434] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:20.773209 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 652903cbbdb042d780e7d71fc2c0ab4d (table default.Table_438d2aad28fd458786870f8dd77671ec [id=0e0c97d7ced7490b86425ea2e2e4530c]) successfully deleted
I20251029 08:52:20.773252 2044 raft_consensus.cc:493] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.773445 2044 raft_consensus.cc:515] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.774199 2044 leader_election.cc:290] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:20.775313 998 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a20f26d0f7874ccb960ffe4d27e854e1
W20251029 08:52:20.776525 996 leader_election.cc:343] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:20.776684 996 leader_election.cc:304] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.776957 2044 raft_consensus.cc:2749] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.777338 1183 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.778793 1178 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.779075 1178 ts_tablet_manager.cc:1403] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20251029 08:52:20.782560 1178 raft_consensus.cc:359] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.782891 1178 raft_consensus.cc:385] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.783011 1178 raft_consensus.cc:740] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.783391 1178 consensus_queue.cc:260] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.741372 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba201a54466240129d82a096f9316242" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.784983 1177 ts_tablet_manager.cc:1403] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.107s user 0.000s sys 0.003s
W20251029 08:52:20.785346 718 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.785519 1179 ts_tablet_manager.cc:1929] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:20.785640 1179 log.cc:1199] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/8a9f0b579ebd4d2db813459611d7f9bf
I20251029 08:52:20.785656 718 leader_election.cc:304] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.785985 1179 ts_tablet_manager.cc:1950] T 8a9f0b579ebd4d2db813459611d7f9bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.786193 2165 raft_consensus.cc:2749] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.752300 901 tablet_service.cc:1505] Processing CreateTablet for tablet 7671e29e48fc478c900959b22a0b98b3 (DEFAULT_TABLE table=default.Table_76bc2e4759364fe488fa32a2afa440a0 [id=3d2aae0d84a04e94aba4073471d8a8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.752408 896 tablet_service.cc:1505] Processing CreateTablet for tablet 8553f806c78f4617bb8d97cdba2243bf (DEFAULT_TABLE table=default.Table_5dc644387108496995ffad8f788f3c24 [id=d38e1320dbd54a379e985e5af8840f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.786835 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7671e29e48fc478c900959b22a0b98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.787057 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8553f806c78f4617bb8d97cdba2243bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.785401 1178 ts_tablet_manager.cc:1434] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 08:52:20.787437 1178 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.789060 1178 tablet_bootstrap.cc:654] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.792675 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 8a9f0b579ebd4d2db813459611d7f9bf (table default.Table_d2fa4bcfb5a240239a27ceb1bebf974c [id=93bd45834be24532b19ed170c43ac0dc]) successfully deleted
W20251029 08:52:20.792966 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 939d7f6b895d460296eb132087ea65af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 939d7f6b895d460296eb132087ea65af already in progress: creating tablet
I20251029 08:52:20.796623 1179 tablet_replica.cc:333] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.796828 1179 raft_consensus.cc:2243] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.796972 1179 raft_consensus.cc:2272] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.797135 1183 tablet_bootstrap.cc:654] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.798025 1177 raft_consensus.cc:359] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.798199 1177 raft_consensus.cc:385] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.797472 1179 ts_tablet_manager.cc:1916] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.798259 1177 raft_consensus.cc:740] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.798473 1177 consensus_queue.cc:260] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.800630 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 7671e29e48fc478c900959b22a0b98b3 (DEFAULT_TABLE table=default.Table_76bc2e4759364fe488fa32a2afa440a0 [id=3d2aae0d84a04e94aba4073471d8a8d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:20.800899 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b0b3b6fa45044c3fb83823ed4f233735 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b3b6fa45044c3fb83823ed4f233735 already in progress: creating tablet
I20251029 08:52:20.800563 896 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd12c9aa01495e83b09e1d82da20ad (DEFAULT_TABLE table=default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.801025 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7671e29e48fc478c900959b22a0b98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.801263 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd12c9aa01495e83b09e1d82da20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.801879 902 tablet_service.cc:1505] Processing CreateTablet for tablet 2b46bb1ed1994c149eb9a1db10b00921 (DEFAULT_TABLE table=default.Table_439c96c50e724a6db3abaac6c5540a36 [id=44c6086367ec4467a96fb0bbb8a749e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.802182 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b46bb1ed1994c149eb9a1db10b00921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.802181 748 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd12c9aa01495e83b09e1d82da20ad (DEFAULT_TABLE table=default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.802573 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd12c9aa01495e83b09e1d82da20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.804283 757 tablet_service.cc:1505] Processing CreateTablet for tablet 7671e29e48fc478c900959b22a0b98b3 (DEFAULT_TABLE table=default.Table_76bc2e4759364fe488fa32a2afa440a0 [id=3d2aae0d84a04e94aba4073471d8a8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.804598 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7671e29e48fc478c900959b22a0b98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.808249 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:20.808426 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd12c9aa01495e83b09e1d82da20ad (DEFAULT_TABLE table=default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.808820 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd12c9aa01495e83b09e1d82da20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.809087 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64a621b6339340e99357f629eeb43501" } modify_external_catalogs: true
I20251029 08:52:20.809185 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.809307 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64a621b6339340e99357f629eeb43501" } modify_external_catalogs: true
I20251029 08:52:20.809525 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d62d1e354d41f49b9ae30dbb6efc0f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:20.809638 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7671e29e48fc478c900959b22a0b98b3 because the tablet was not found. No further retry: Not found: Tablet not found: 7671e29e48fc478c900959b22a0b98b3
I20251029 08:52:20.810110 1176 tablet_replica.cc:333] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.810300 1176 raft_consensus.cc:2243] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.810433 1176 raft_consensus.cc:2272] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.810868 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485c9946f3f14a2aab11f4cdaa70a5b6 already in progress: creating tablet
I20251029 08:52:20.811475 1176 ts_tablet_manager.cc:1916] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.817999 2156 raft_consensus.cc:493] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.818150 2156 raft_consensus.cc:515] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.818684 2156 leader_election.cc:290] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:20.820816 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:20.821349 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e2be77182e4259b76d20cae4e9e478" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.822469 861 leader_election.cc:304] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:20.823009 2199 raft_consensus.cc:2749] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.823771 1177 ts_tablet_manager.cc:1434] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.039s user 0.000s sys 0.001s
I20251029 08:52:20.826014 1176 ts_tablet_manager.cc:1929] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.826128 1176 log.cc:1199] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/43c3074caee74069a8a7f1b87721d726
I20251029 08:52:20.826537 1179 ts_tablet_manager.cc:1929] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.826646 1179 log.cc:1199] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/787d6d7f1870450fa21cb9cba34e72fe
I20251029 08:52:20.826901 1179 ts_tablet_manager.cc:1950] T 787d6d7f1870450fa21cb9cba34e72fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.827781 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 787d6d7f1870450fa21cb9cba34e72fe (table default.Table_9a1d9f17776045feba0e8b4104b2fd24 [id=d3193569dcd746cdb13098748d380ccc]) successfully deleted
I20251029 08:52:20.828230 1177 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:20.829974 1177 tablet_bootstrap.cc:654] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.830928 1179 tablet_replica.cc:333] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.831125 1179 raft_consensus.cc:2243] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.831262 1179 raft_consensus.cc:2272] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.831748 1179 ts_tablet_manager.cc:1916] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.826440 1176 ts_tablet_manager.cc:1950] T 43c3074caee74069a8a7f1b87721d726 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.836810 757 tablet_service.cc:1505] Processing CreateTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.837278 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d62d1e354d41f49b9ae30dbb6efc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.842161 2066 hybrid_clock.cc:648] HybridClock initialized: now 1761727940841949 us; error 216 us; skew 500 ppm
I20251029 08:52:20.842741 2066 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 08:52:20.844208 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9da1e9208d495185f625c5015dc23c (DEFAULT_TABLE table=default.Table_022219d62443456187fdfdf9562de3f7 [id=2321dd7fd5924c8ea3871b09c3259ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.844745 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9da1e9208d495185f625c5015dc23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.845507 2044 raft_consensus.cc:493] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.846324 2044 raft_consensus.cc:515] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.846638 1176 tablet_replica.cc:333] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.846820 1176 raft_consensus.cc:2243] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.846951 1176 raft_consensus.cc:2272] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.846956 2044 leader_election.cc:290] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.847209 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 43c3074caee74069a8a7f1b87721d726 (table default.Table_8c5b5a48612f4ed4a391c6cd25320eea [id=80e7ea583ee14114878007a237b185c8]) successfully deleted
I20251029 08:52:20.847774 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:20.847942 784 raft_consensus.cc:2468] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:20.848176 1176 ts_tablet_manager.cc:1916] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.848459 757 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9da1e9208d495185f625c5015dc23c (DEFAULT_TABLE table=default.Table_022219d62443456187fdfdf9562de3f7 [id=2321dd7fd5924c8ea3871b09c3259ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.848634 998 leader_election.cc:304] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:20.848802 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9da1e9208d495185f625c5015dc23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.848990 2044 raft_consensus.cc:2804] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:20.849066 2044 raft_consensus.cc:493] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:20.849140 2044 raft_consensus.cc:3060] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:20.852705 1179 ts_tablet_manager.cc:1929] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.852762 2044 raft_consensus.cc:515] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.854205 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:20.854423 784 raft_consensus.cc:3060] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:20.856760 1178 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.856884 1178 ts_tablet_manager.cc:1403] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20251029 08:52:20.857656 1178 raft_consensus.cc:359] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.857796 1178 raft_consensus.cc:385] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.857837 1178 raft_consensus.cc:740] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.858631 784 raft_consensus.cc:2468] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:20.859501 998 leader_election.cc:304] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:20.859867 2197 raft_consensus.cc:2804] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:20.860862 2044 leader_election.cc:290] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.852815 1179 log.cc:1199] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2c706343d44a4807b1ea60c1b1e1ea49
I20251029 08:52:20.861734 1179 ts_tablet_manager.cc:1950] T 2c706343d44a4807b1ea60c1b1e1ea49 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.862967 1179 tablet_replica.cc:333] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:20.862949 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2c706343d44a4807b1ea60c1b1e1ea49 (table default.Table_d71bb6e219494f90897bd2f8844f599e [id=ddbca021ab254f55b10a491fa02fdbcc]) successfully deleted
I20251029 08:52:20.863312 1179 raft_consensus.cc:2243] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.863508 1179 raft_consensus.cc:2272] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.863926 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:40435, user_credentials={real_user=slave}} blocked reactor thread for 52716.1us
I20251029 08:52:20.865094 2066 webserver.cc:492] Webserver started at http://127.0.104.62:37745/ using document root <none> and password file <none>
I20251029 08:52:20.866281 1178 consensus_queue.cc:260] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.866711 2066 fs_manager.cc:362] Metadata directory not provided
I20251029 08:52:20.866955 2066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 08:52:20.870304 1179 ts_tablet_manager.cc:1916] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.871699 1178 ts_tablet_manager.cc:1434] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251029 08:52:20.844208 893 tablet_service.cc:1505] Processing CreateTablet for tablet b582dfeab662456ba4730f5c80455d83 (DEFAULT_TABLE table=default.Table_f760df7015424091acf5d9b6b39bfbd3 [id=ff4e798002804cfeaba371703af37228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.876505 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b582dfeab662456ba4730f5c80455d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.880465 1178 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.844532 887 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9da1e9208d495185f625c5015dc23c (DEFAULT_TABLE table=default.Table_022219d62443456187fdfdf9562de3f7 [id=2321dd7fd5924c8ea3871b09c3259ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.852553 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.860963 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1da3e7940352460c9b2bb060174829c9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:20.883809 996 leader_election.cc:343] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:20.884202 996 leader_election.cc:343] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.884258 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9da1e9208d495185f625c5015dc23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.883781 1183 tablet_bootstrap.cc:492] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:20.884564 1183 ts_tablet_manager.cc:1403] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.003s
I20251029 08:52:20.885311 1183 raft_consensus.cc:359] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.885490 1183 raft_consensus.cc:385] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.885541 1183 raft_consensus.cc:740] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.885741 1183 consensus_queue.cc:260] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.886632 2197 raft_consensus.cc:697] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:20.882747 1176 ts_tablet_manager.cc:1929] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.886937 1176 log.cc:1199] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:20.886909 2197 consensus_queue.cc:237] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:20.887079 1183 ts_tablet_manager.cc:1434] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:20.887429 1176 ts_tablet_manager.cc:1950] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.887859 1183 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:20.888644 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (table default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]) successfully deleted
I20251029 08:52:20.888995 1176 tablet_replica.cc:333] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:20.889204 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 65b20d53d6214bdc81ca205dbf922d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65b20d53d6214bdc81ca205dbf922d7b already in progress: creating tablet
I20251029 08:52:20.889228 1176 raft_consensus.cc:2243] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:20.889454 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4e213713757949caa3f504f70026b7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e213713757949caa3f504f70026b7b5 already in progress: creating tablet
I20251029 08:52:20.889513 1176 raft_consensus.cc:2272] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.889699 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a712185fe4aa4f8faf1ce2df229c11ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a712185fe4aa4f8faf1ce2df229c11ee already in progress: creating tablet
I20251029 08:52:20.890159 1176 ts_tablet_manager.cc:1916] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.890298 1183 tablet_bootstrap.cc:654] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.876116 1974 ts_tablet_manager.cc:1929] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.891103 1974 log.cc:1199] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/fd25eeb89e574297a92d8e978a79d768
I20251029 08:52:20.891492 1974 ts_tablet_manager.cc:1950] T fd25eeb89e574297a92d8e978a79d768 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:20.892123 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet fd25eeb89e574297a92d8e978a79d768 (table default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]) successfully deleted
I20251029 08:52:20.894495 901 tablet_service.cc:1505] Processing CreateTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.895251 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d62d1e354d41f49b9ae30dbb6efc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.896412 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 31bed64a27164baa8177b87b54b5947b
I20251029 08:52:20.896912 588 catalog_manager.cc:6719] ReplaceTablet: tablet 31bed64a27164baa8177b87b54b5947b deleted and replaced by tablet 5e3782c226c14143aec56367d1a04996
I20251029 08:52:20.897115 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 31bed64a27164baa8177b87b54b5947b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e3782c226c14143aec56367d1a04996) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.898164 1035 tablet_service.cc:1505] Processing CreateTablet for tablet b582dfeab662456ba4730f5c80455d83 (DEFAULT_TABLE table=default.Table_f760df7015424091acf5d9b6b39bfbd3 [id=ff4e798002804cfeaba371703af37228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.898293 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 31bed64a27164baa8177b87b54b5947b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e3782c226c14143aec56367d1a04996) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.898530 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b582dfeab662456ba4730f5c80455d83. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:20.899010 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a712185fe4aa4f8faf1ce2df229c11ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a712185fe4aa4f8faf1ce2df229c11ee already in progress: creating tablet
I20251029 08:52:20.899291 1974 tablet_replica.cc:333] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:20.899488 1974 raft_consensus.cc:2243] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:20.899649 1974 raft_consensus.cc:2272] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:20.900717 1974 ts_tablet_manager.cc:1916] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:20.901180 893 tablet_service.cc:1505] Processing CreateTablet for tablet 8664571228fa47a7b6f79f9d62c2be11 (DEFAULT_TABLE table=default.Table_9d51a88fcb8d479f8b0e2d77e02cf90a [id=ba9ecf1bc9e6487cb9d25292b00ea424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.901554 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8664571228fa47a7b6f79f9d62c2be11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.902904 2066 fs_manager.cc:714] Time spent opening directory manager: real 0.033s user 0.005s sys 0.000s
I20251029 08:52:20.905481 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 8664571228fa47a7b6f79f9d62c2be11 (DEFAULT_TABLE table=default.Table_9d51a88fcb8d479f8b0e2d77e02cf90a [id=ba9ecf1bc9e6487cb9d25292b00ea424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.905851 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8664571228fa47a7b6f79f9d62c2be11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.905927 896 tablet_service.cc:1505] Processing CreateTablet for tablet 72d0540040614513a4518ac983d14a94 (DEFAULT_TABLE table=default.Table_fe464f17db3547f393d347ee7c8643c2 [id=e2c5bc8265cc4867956909cb0fcebf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.906241 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d0540040614513a4518ac983d14a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.884236 1178 tablet_bootstrap.cc:654] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.904735 748 tablet_service.cc:1505] Processing CreateTablet for tablet b582dfeab662456ba4730f5c80455d83 (DEFAULT_TABLE table=default.Table_f760df7015424091acf5d9b6b39bfbd3 [id=ff4e798002804cfeaba371703af37228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.910154 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b582dfeab662456ba4730f5c80455d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.906832 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 31bed64a27164baa8177b87b54b5947b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e3782c226c14143aec56367d1a04996) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.908285 758 tablet_service.cc:1505] Processing CreateTablet for tablet 8664571228fa47a7b6f79f9d62c2be11 (DEFAULT_TABLE table=default.Table_9d51a88fcb8d479f8b0e2d77e02cf90a [id=ba9ecf1bc9e6487cb9d25292b00ea424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.916369 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8664571228fa47a7b6f79f9d62c2be11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.919204 757 tablet_service.cc:1505] Processing CreateTablet for tablet 72d0540040614513a4518ac983d14a94 (DEFAULT_TABLE table=default.Table_fe464f17db3547f393d347ee7c8643c2 [id=e2c5bc8265cc4867956909cb0fcebf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.919533 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d0540040614513a4518ac983d14a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.922991 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 72d0540040614513a4518ac983d14a94 (DEFAULT_TABLE table=default.Table_fe464f17db3547f393d347ee7c8643c2 [id=e2c5bc8265cc4867956909cb0fcebf55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.923434 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d0540040614513a4518ac983d14a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.926946 2257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 08:52:20.928885 1178 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.929028 1178 ts_tablet_manager.cc:1403] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20251029 08:52:20.929778 1178 raft_consensus.cc:359] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.929832 902 tablet_service.cc:1505] Processing CreateTablet for tablet d8623b1f73194d3084b4ac04568858e1 (DEFAULT_TABLE table=default.Table_5d8159a04f9f454c8d3d5b4472097608 [id=0f0e2035a9ea46088af78daf169b2ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.929950 1178 raft_consensus.cc:385] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.930002 1178 raft_consensus.cc:740] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.930222 1178 consensus_queue.cc:260] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.930681 1178 ts_tablet_manager.cc:1434] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:20.931349 2165 raft_consensus.cc:493] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.931491 2165 raft_consensus.cc:515] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:20.931536 1035 tablet_service.cc:1505] Processing CreateTablet for tablet d8623b1f73194d3084b4ac04568858e1 (DEFAULT_TABLE table=default.Table_5d8159a04f9f454c8d3d5b4472097608 [id=0f0e2035a9ea46088af78daf169b2ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.931910 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8623b1f73194d3084b4ac04568858e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.932067 2165 leader_election.cc:290] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:20.932682 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa4f97075b3941aaa7cda3f9413e867f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:20.933228 720 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.933295 757 tablet_service.cc:1505] Processing CreateTablet for tablet d8623b1f73194d3084b4ac04568858e1 (DEFAULT_TABLE table=default.Table_5d8159a04f9f454c8d3d5b4472097608 [id=0f0e2035a9ea46088af78daf169b2ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.932943 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa4f97075b3941aaa7cda3f9413e867f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:20.935108 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8623b1f73194d3084b4ac04568858e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:20.935572 718 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.935678 718 leader_election.cc:304] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:20.936098 2165 raft_consensus.cc:2749] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:20.939901 1176 ts_tablet_manager.cc:1929] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.940138 892 tablet_service.cc:1505] Processing CreateTablet for tablet 55ec37d15bb64edf8d6552ee068d3573 (DEFAULT_TABLE table=default.Table_c2da7feabcf14a2f9d60e558d438633f [id=28f26a20adc449c3bd09a6229e43685b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.943873 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 55ec37d15bb64edf8d6552ee068d3573 (DEFAULT_TABLE table=default.Table_c2da7feabcf14a2f9d60e558d438633f [id=28f26a20adc449c3bd09a6229e43685b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.944305 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ec37d15bb64edf8d6552ee068d3573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.950318 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:20.950676 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:20.951021 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 026eef7a3cba40d0a2c090485cea55e0 (table default.Table_5471f52640fd473e90327f739bd8223d [id=559cd9144082414dade2ac5d73a7057c]). Expected version 2 got 0
I20251029 08:52:20.951381 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_34c608523fd44399b9c71a365266edd6 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 4 got 0
I20251029 08:52:20.955749 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:20.956655 2066 fs_manager.cc:730] Time spent opening block manager: real 0.052s user 0.003s sys 0.001s
I20251029 08:52:20.957469 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 084a659932ec4996a6d6b160d67efdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.956890 2066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
uuid: "f718bd0f00004f20a89d028a2fa8e6b9"
format_stamp: "Formatted at 2025-10-29 08:51:58 on dist-test-slave-3g17"
I20251029 08:52:20.958305 887 tablet_service.cc:1552] Processing DeleteTablet for tablet ea13e9ea2c11479ea86e3fd7f2a9e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:20.958197 2066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 08:52:20.961946 588 catalog_manager.cc:5649] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:20.962569 887 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1aa4bf9b47439eaba24f60e955f14e (DEFAULT_TABLE table=default.Table_36384ce4abb64dd886df8435345c4a46 [id=af1edc004ba1430b9edac3ef614baa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.964136 1176 log.cc:1199] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4aa0ff14e19b4ad7a8b33796481f0f3f
I20251029 08:52:20.964596 1176 ts_tablet_manager.cc:1950] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:20.967235 1974 ts_tablet_manager.cc:1929] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.967911 1176 tablet_replica.cc:333] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:20.952194 2165 raft_consensus.cc:493] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:20.953014 757 tablet_service.cc:1505] Processing CreateTablet for tablet 55ec37d15bb64edf8d6552ee068d3573 (DEFAULT_TABLE table=default.Table_c2da7feabcf14a2f9d60e558d438633f [id=28f26a20adc449c3bd09a6229e43685b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.975636 2165 raft_consensus.cc:515] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.975865 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ec37d15bb64edf8d6552ee068d3573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.976368 2165 leader_election.cc:290] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:20.962728 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0195d09f103745aca4e75dfbeb05f244" } modify_external_catalogs: true
I20251029 08:52:20.940948 1178 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.976986 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0195d09f103745aca4e75dfbeb05f244" } modify_external_catalogs: true
I20251029 08:52:20.977496 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:20.977927 1176 raft_consensus.cc:2243] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:20.978106 720 leader_election.cc:343] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:20.978649 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:20.978960 1178 tablet_bootstrap.cc:654] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:20.978117 1176 raft_consensus.cc:2272] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:20.980721 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e039e5f7f941410cb42a21502a1844a4 consensus state has no leader
I20251029 08:52:20.968101 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (table default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]) successfully deleted
W20251029 08:52:20.981439 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57c0ad83e3f54150850b6617a1781382 consensus state has no leader
I20251029 08:52:20.981529 1176 ts_tablet_manager.cc:1916] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:20.983814 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d08838d55e248ba99e00532bb82b65f consensus state has no leader
I20251029 08:52:20.986212 1179 ts_tablet_manager.cc:1929] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:20.986371 1179 log.cc:1199] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:20.986719 1179 ts_tablet_manager.cc:1950] T 2723cdd513ee4e8ca6773b38f785b09e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:20.986678 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_74466c51bbac4f8a9c086680a8eda1d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:20.987259 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74466c51bbac4f8a9c086680a8eda1d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.987741 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2723cdd513ee4e8ca6773b38f785b09e (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:20.988582 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_09897b5139274e6fb00dd3a14c51db4d" } modify_external_catalogs: true
I20251029 08:52:20.988783 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_09897b5139274e6fb00dd3a14c51db4d" } modify_external_catalogs: true
I20251029 08:52:20.988927 1178 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:20.989053 1178 ts_tablet_manager.cc:1403] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.001s
I20251029 08:52:20.989584 757 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1aa4bf9b47439eaba24f60e955f14e (DEFAULT_TABLE table=default.Table_36384ce4abb64dd886df8435345c4a46 [id=af1edc004ba1430b9edac3ef614baa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.989801 1178 raft_consensus.cc:359] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.989945 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1aa4bf9b47439eaba24f60e955f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.989960 1178 raft_consensus.cc:385] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:20.990041 1178 raft_consensus.cc:740] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:20.990129 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_15cd2138e80e429db3004528a89ec2bf" } modify_external_catalogs: true
I20251029 08:52:20.990298 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_15cd2138e80e429db3004528a89ec2bf" } modify_external_catalogs: true
I20251029 08:52:20.990262 1178 consensus_queue.cc:260] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:20.992478 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 900e8494364247289fa4208d81dfe3bb (DEFAULT_TABLE table=default.Table_6e7b5c67ba3644c4be9ef7129ea50504 [id=cceccaff4bd944b6a6e179009b147a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.992566 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fcaa3fa779824b38887ae001623ba67b
I20251029 08:52:20.993031 597 catalog_manager.cc:6719] ReplaceTablet: tablet fcaa3fa779824b38887ae001623ba67b deleted and replaced by tablet 919e5494b84343639a2b9b81b05f1c2c
I20251029 08:52:20.993136 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.993345 1178 ts_tablet_manager.cc:1434] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:20.993626 1178 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:20.993937 748 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:20.994081 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1aa4bf9b47439eaba24f60e955f14e (DEFAULT_TABLE table=default.Table_36384ce4abb64dd886df8435345c4a46 [id=af1edc004ba1430b9edac3ef614baa57]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:20.995715 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85b300f17a704f7dba1aa3724fcc2de7 consensus state has no leader
I20251029 08:52:20.995977 1044 tablet_service.cc:1505] Processing CreateTablet for tablet cac7c7cf4679477bab723eec86c97ee9 (DEFAULT_TABLE table=default.Table_0195d09f103745aca4e75dfbeb05f244 [id=9befe3b7b37c40f4a26636e958e52b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.997200 748 tablet_service.cc:1505] Processing CreateTablet for tablet cac7c7cf4679477bab723eec86c97ee9 (DEFAULT_TABLE table=default.Table_0195d09f103745aca4e75dfbeb05f244 [id=9befe3b7b37c40f4a26636e958e52b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:20.997543 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7c7cf4679477bab723eec86c97ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.999049 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 663d6587a5e14519afbb0f94d691d1b4 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:20.999821 1178 tablet_bootstrap.cc:654] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.000854 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.001050 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.001211 753 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.001322 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.002094 758 tablet_service.cc:1505] Processing CreateTablet for tablet 900e8494364247289fa4208d81dfe3bb (DEFAULT_TABLE table=default.Table_6e7b5c67ba3644c4be9ef7129ea50504 [id=cceccaff4bd944b6a6e179009b147a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.002646 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900e8494364247289fa4208d81dfe3bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.003800 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d9cabbbac8254a949750244e1d6991eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9cabbbac8254a949750244e1d6991eb already in progress: creating tablet
I20251029 08:52:20.974457 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ec37d15bb64edf8d6552ee068d3573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.977229 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.008092 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900e8494364247289fa4208d81dfe3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:20.981182 924 raft_consensus.cc:3060] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.010058 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 90360621ce124475af2f12060471f63c
I20251029 08:52:20.992146 891 tablet_service.cc:1505] Processing CreateTablet for tablet cac7c7cf4679477bab723eec86c97ee9 (DEFAULT_TABLE table=default.Table_0195d09f103745aca4e75dfbeb05f244 [id=9befe3b7b37c40f4a26636e958e52b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.011823 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7c7cf4679477bab723eec86c97ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.012642 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cc7e3f702f9a407abf4fe7c7be513c67
I20251029 08:52:21.014387 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1949c3fc4543485b88fb4788719aed4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.014982 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1949c3fc4543485b88fb4788719aed4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:20.993541 889 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.016212 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7ee09f96373f4bb19ffe77904d76cb95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.016734 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ee09f96373f4bb19ffe77904d76cb95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.017055 2197 raft_consensus.cc:493] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.017197 2197 raft_consensus.cc:515] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.017735 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 90360621ce124475af2f12060471f63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.017823 2197 leader_election.cc:290] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.019405 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 667c71a8dfd54c05ab56e845653c0c9a (table default.Table_c9c0266a3d25476cae75216163270a5f [id=3b0c410e5d1e4d33ab2f50cff45fb590]). Expected version 1 got 0
I20251029 08:52:21.019685 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:21.020419 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:20.996420 895 tablet_service.cc:1505] Processing CreateTablet for tablet 475f5bfe13b8440993cc62fcc9b29f8d (DEFAULT_TABLE table=default.Table_a10a39719edd4acfb78ffc1e3c43d550 [id=775d6e5564204aa1a82ce68f4559ca58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.021054 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8c17839d47d445acb4e7f4d30d5850dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.021184 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475f5bfe13b8440993cc62fcc9b29f8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.022091 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c17839d47d445acb4e7f4d30d5850dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.023396 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f3b73a492eca4e8fa1f1365cbc3518af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.023835 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b73a492eca4e8fa1f1365cbc3518af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.025702 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 90360621ce124475af2f12060471f63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:21.025950 996 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 8991ea78321545be85d5c480841e4751
I20251029 08:52:21.026170 2197 raft_consensus.cc:493] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.026310 2197 raft_consensus.cc:515] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.026379 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2a5ef8abeebf415084663ef00f8d942b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.026806 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a5ef8abeebf415084663ef00f8d942b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.026901 2197 leader_election.cc:290] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.027694 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea13e9ea2c11479ea86e3fd7f2a9e59c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.027870 922 raft_consensus.cc:2468] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:21.028272 996 leader_election.cc:304] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:21.028541 2197 raft_consensus.cc:2804] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.028626 2197 raft_consensus.cc:493] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.028687 2197 raft_consensus.cc:3060] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.031373 2197 raft_consensus.cc:515] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.033300 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 90360621ce124475af2f12060471f63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:21.033493 718 leader_election.cc:343] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.033632 718 leader_election.cc:304] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.033684 748 tablet_service.cc:1552] Processing DeleteTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.033869 2165 raft_consensus.cc:2749] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.034165 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7c7cf4679477bab723eec86c97ee9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.034552 998 leader_election.cc:343] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ea13e9ea2c11479ea86e3fd7f2a9e59c
W20251029 08:52:21.038242 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0d08838d55e248ba99e00532bb82b65f (table default.Table_c30d9ab9918b4baaa0a6d733432364c1 [id=7aa97e70d17b4fcb89af7f7028148a03], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0d08838d55e248ba99e00532bb82b65f consensus state has no leader
I20251029 08:52:21.040396 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1aa4bf9b47439eaba24f60e955f14e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.040436 998 leader_election.cc:343] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 8991ea78321545be85d5c480841e4751
I20251029 08:52:21.040620 998 leader_election.cc:304] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:21.041056 998 leader_election.cc:343] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ea13e9ea2c11479ea86e3fd7f2a9e59c
I20251029 08:52:21.041316 2249 raft_consensus.cc:2749] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.044658 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cac7c7cf4679477bab723eec86c97ee9
I20251029 08:52:21.045353 889 tablet_service.cc:1552] Processing DeleteTablet for tablet cac7c7cf4679477bab723eec86c97ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.046010 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet cac7c7cf4679477bab723eec86c97ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.047245 748 tablet_service.cc:1552] Processing DeleteTablet for tablet cac7c7cf4679477bab723eec86c97ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.047777 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 791) took 1074 ms. Trace:
I20251029 08:52:21.047896 1179 rpcz_store.cc:276] 1029 08:52:19.973071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:20.071825 (+ 98754us) service_pool.cc:225] Handling call
1029 08:52:21.004094 (+932269us) ts_tablet_manager.cc:1151] Deleting tablet ff1b7181e52041288c45b053124bd844
1029 08:52:21.033877 (+ 29783us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:21.047759 (+ 13882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931989}
I20251029 08:52:21.048008 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea13e9ea2c11479ea86e3fd7f2a9e59c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:21.048226 922 raft_consensus.cc:3060] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:21.048480 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ff1b7181e52041288c45b053124bd844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b7181e52041288c45b053124bd844 already in progress: creating tablet
I20251029 08:52:21.048871 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 790) took 1078 ms. Trace:
I20251029 08:52:21.048938 1179 rpcz_store.cc:276] 1029 08:52:19.970932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:20.145458 (+174526us) service_pool.cc:225] Handling call
1029 08:52:21.048758 (+903300us) ts_tablet_manager.cc:1151] Deleting tablet a013257089774ba8b1f535e5f94ac1bb
1029 08:52:21.048768 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:21.048856 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":874382}
W20251029 08:52:21.049357 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a013257089774ba8b1f535e5f94ac1bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a013257089774ba8b1f535e5f94ac1bb already in progress: creating tablet
I20251029 08:52:21.049721 1179 tablet_replica.cc:333] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.049901 1179 raft_consensus.cc:2243] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.050029 1179 raft_consensus.cc:2272] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.050513 1179 ts_tablet_manager.cc:1916] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.050952 922 raft_consensus.cc:2468] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:21.051563 996 leader_election.cc:304] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:21.051841 2249 raft_consensus.cc:2804] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.051945 2249 raft_consensus.cc:697] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:21.052203 2249 consensus_queue.cc:237] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.053653 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4028cb8d8c3e48c48ef5c82e7febdd54" } new_table_name: "default.Table_d707b992dd69414897e525d86fb4b002" modify_external_catalogs: true
I20251029 08:52:21.056686 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a08630cbc401467a9dde1ce1e198cef7
I20251029 08:52:21.057297 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 43c3074caee74069a8a7f1b87721d726 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:21.057828 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1a22395ba73c42eab774fb043c38da7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.057978 2197 leader_election.cc:290] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:21.058424 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a22395ba73c42eab774fb043c38da7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.059693 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 34729c3f97a44713a352bde297a80027
I20251029 08:52:21.063262 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0d4cf693087a4d308c9e97704e60754d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.063396 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 34729c3f97a44713a352bde297a80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:21.063943 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d4cf693087a4d308c9e97704e60754d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.063931 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 34729c3f97a44713a352bde297a80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.064193 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet a08630cbc401467a9dde1ce1e198cef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.064414 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8623b1f73194d3084b4ac04568858e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.065912 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:06 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.066032 1178 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.066174 1178 ts_tablet_manager.cc:1403] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.010s
I20251029 08:52:21.066321 2249 raft_consensus.cc:493] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.066468 2249 raft_consensus.cc:515] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.067013 2249 leader_election.cc:290] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.067927 1178 raft_consensus.cc:359] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.068145 1178 raft_consensus.cc:385] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.068197 1178 raft_consensus.cc:740] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.068427 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.068460 1178 consensus_queue.cc:260] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.069490 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c515944ecf4228a7cca63cfd3565d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:21.069921 998 leader_election.cc:304] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.070377 2249 raft_consensus.cc:2749] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.070760 1178 ts_tablet_manager.cc:1434] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 08:52:21.070818 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1aa4bf9b47439eaba24f60e955f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.071048 1178 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.073993 1179 ts_tablet_manager.cc:1929] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.076152 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a08630cbc401467a9dde1ce1e198cef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.076167 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]). Expected version 1 got 0
I20251029 08:52:21.076498 891 tablet_service.cc:1552] Processing DeleteTablet for tablet a08630cbc401467a9dde1ce1e198cef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.076857 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 34729c3f97a44713a352bde297a80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.000850 901 tablet_service.cc:1505] Processing CreateTablet for tablet 900e8494364247289fa4208d81dfe3bb (DEFAULT_TABLE table=default.Table_6e7b5c67ba3644c4be9ef7129ea50504 [id=cceccaff4bd944b6a6e179009b147a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.079566 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900e8494364247289fa4208d81dfe3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.080484 1179 log.cc:1199] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c572c68b32d443b2a019e1f4bf2d82fe
I20251029 08:52:21.080787 2165 raft_consensus.cc:493] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.080920 2165 raft_consensus.cc:515] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.081048 1179 ts_tablet_manager.cc:1950] T c572c68b32d443b2a019e1f4bf2d82fe P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.081529 2165 leader_election.cc:290] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.082558 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627bc8ff99394fa2abcca15d62e5ea10" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.082592 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627bc8ff99394fa2abcca15d62e5ea10" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.083221 720 leader_election.cc:343] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.083388 893 tablet_service.cc:1552] Processing DeleteTablet for tablet cc7e3f702f9a407abf4fe7c7be513c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:21.083681 718 leader_election.cc:343] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.083808 718 leader_election.cc:304] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.084045 2165 raft_consensus.cc:2749] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.084887 1178 tablet_bootstrap.cc:654] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.085394 1179 tablet_replica.cc:333] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.085536 1974 log.cc:1199] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/449ffc678b874871a672c3059235de6a
I20251029 08:52:21.085788 1179 raft_consensus.cc:2243] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.085947 1179 raft_consensus.cc:2272] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.086426 1179 ts_tablet_manager.cc:1916] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.087177 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c572c68b32d443b2a019e1f4bf2d82fe (table default.Table_d1dd83e945754532a1f9694d8c9d2e0c [id=95f0b457b6e345cd96cce24c4bc1d82a]) successfully deleted
W20251029 08:52:21.087522 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485c9946f3f14a2aab11f4cdaa70a5b6 already in progress: creating tablet
W20251029 08:52:21.087777 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7671e29e48fc478c900959b22a0b98b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7671e29e48fc478c900959b22a0b98b3 already in progress: creating tablet
W20251029 08:52:21.088076 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 65b20d53d6214bdc81ca205dbf922d7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65b20d53d6214bdc81ca205dbf922d7b already in progress: creating tablet
W20251029 08:52:21.088326 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 already in progress: creating tablet
I20251029 08:52:21.088141 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1636c78f17a549d7b97dfacee192f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.088217 1974 ts_tablet_manager.cc:1950] T 449ffc678b874871a672c3059235de6a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.090134 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 475f5bfe13b8440993cc62fcc9b29f8d (DEFAULT_TABLE table=default.Table_a10a39719edd4acfb78ffc1e3c43d550 [id=775d6e5564204aa1a82ce68f4559ca58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.090596 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475f5bfe13b8440993cc62fcc9b29f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.090833 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 449ffc678b874871a672c3059235de6a (table default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]) successfully deleted
I20251029 08:52:21.091878 1177 tablet_bootstrap.cc:492] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.092005 1177 ts_tablet_manager.cc:1403] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.265s user 0.002s sys 0.002s
I20251029 08:52:21.092722 1177 raft_consensus.cc:359] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.092909 1177 raft_consensus.cc:385] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.092964 1177 raft_consensus.cc:740] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.093180 1177 consensus_queue.cc:260] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.093653 1177 ts_tablet_manager.cc:1434] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:21.094079 1177 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.095557 1177 tablet_bootstrap.cc:654] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.097000 1176 ts_tablet_manager.cc:1929] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.097132 1176 log.cc:1199] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:21.097472 1176 ts_tablet_manager.cc:1950] T 7be4bcf9e3324bd884e946fae20648fd P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.098773 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7be4bcf9e3324bd884e946fae20648fd (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
I20251029 08:52:21.100734 757 tablet_service.cc:1505] Processing CreateTablet for tablet 475f5bfe13b8440993cc62fcc9b29f8d (DEFAULT_TABLE table=default.Table_a10a39719edd4acfb78ffc1e3c43d550 [id=775d6e5564204aa1a82ce68f4559ca58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.101120 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475f5bfe13b8440993cc62fcc9b29f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.109750 758 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.110512 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.111768 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.116202 1974 tablet_replica.cc:333] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.116418 1974 raft_consensus.cc:2243] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.116632 1974 raft_consensus.cc:2272] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.113461 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 31bed64a27164baa8177b87b54b5947b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31bed64a27164baa8177b87b54b5947b already in progress: creating tablet
I20251029 08:52:21.118451 893 tablet_service.cc:1505] Processing CreateTablet for tablet 33f1feee1c4d4a98bdb23ac66ac337aa (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.115196 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_991230beb13844e8b60b5cdcc634a480"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.119154 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f1feee1c4d4a98bdb23ac66ac337aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.119189 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_991230beb13844e8b60b5cdcc634a480 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.121569 2197 raft_consensus.cc:493] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.121742 2197 raft_consensus.cc:515] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.123208 758 tablet_service.cc:1505] Processing CreateTablet for tablet 33f1feee1c4d4a98bdb23ac66ac337aa (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:21.123425 998 leader_election.cc:343] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:21.123772 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f1feee1c4d4a98bdb23ac66ac337aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.124159 2197 leader_election.cc:290] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.124433 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "084a659932ec4996a6d6b160d67efdd0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.124663 922 raft_consensus.cc:2468] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:21.124959 2066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 08:52:21.125211 996 leader_election.cc:304] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:21.126025 2197 raft_consensus.cc:2804] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.126091 1179 ts_tablet_manager.cc:1929] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.126402 1179 log.cc:1199] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a20f26d0f7874ccb960ffe4d27e854e1
I20251029 08:52:21.126781 1179 ts_tablet_manager.cc:1950] T a20f26d0f7874ccb960ffe4d27e854e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.126114 2197 raft_consensus.cc:493] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.128180 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a20f26d0f7874ccb960ffe4d27e854e1 (table default.Table_d7165655d9634a7bb270da6738b6c0c5 [id=a68d820a7a424238b30b123cf0adc2a7]) successfully deleted
I20251029 08:52:21.128418 2197 raft_consensus.cc:3060] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.129454 1183 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.129599 1183 ts_tablet_manager.cc:1403] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.242s user 0.005s sys 0.001s
I20251029 08:52:21.130384 1183 raft_consensus.cc:359] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.130563 1183 raft_consensus.cc:385] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.130627 1183 raft_consensus.cc:740] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.131817 2197 raft_consensus.cc:515] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.133281 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "084a659932ec4996a6d6b160d67efdd0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:21.133385 998 leader_election.cc:343] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:21.133580 922 raft_consensus.cc:3060] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.136549 922 raft_consensus.cc:2468] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
W20251029 08:52:21.136629 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4e213713757949caa3f504f70026b7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e213713757949caa3f504f70026b7b5 already in progress: creating tablet
I20251029 08:52:21.136865 2197 leader_election.cc:290] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.137126 1179 tablet_replica.cc:333] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.137181 996 leader_election.cc:304] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:21.137460 1179 raft_consensus.cc:2243] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.137657 2197 raft_consensus.cc:2804] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.139232 2066 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 08:52:21.137714 1179 raft_consensus.cc:2272] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.138381 2197 raft_consensus.cc:697] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:21.140725 2197 consensus_queue.cc:237] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.141475 1974 ts_tablet_manager.cc:1916] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.141458 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 33f1feee1c4d4a98bdb23ac66ac337aa (DEFAULT_TABLE table=default.Table_4be9988045844a5b90664d5b0a9238af [id=3e75a44ecf8c4413ba6a5ca3e376067d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.141042 1179 ts_tablet_manager.cc:1916] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.141916 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f1feee1c4d4a98bdb23ac66ac337aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.142355 2066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 08:52:21.143637 895 tablet_service.cc:1505] Processing CreateTablet for tablet cf71cd77855841b5a914a3056a130112 (DEFAULT_TABLE table=default.Table_c312fa246a2c4435b6a2c1fa67420ec0 [id=15d501a0104b4a1ab806d57b6cc32caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.144090 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf71cd77855841b5a914a3056a130112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.144104 1177 tablet_bootstrap.cc:492] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.144330 1177 ts_tablet_manager.cc:1403] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.005s
I20251029 08:52:21.145092 1177 raft_consensus.cc:359] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.145259 1177 raft_consensus.cc:385] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.145311 1177 raft_consensus.cc:740] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.145507 1177 consensus_queue.cc:260] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.145936 1177 ts_tablet_manager.cc:1434] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:21.146188 1177 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.147262 2249 raft_consensus.cc:493] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.147395 2249 raft_consensus.cc:515] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.148007 2249 leader_election.cc:290] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.148710 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:21.148898 780 raft_consensus.cc:2468] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:21.149300 998 leader_election.cc:304] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:21.149891 2249 raft_consensus.cc:2804] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.149971 2249 raft_consensus.cc:493] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.150032 2249 raft_consensus.cc:3060] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.153017 2249 raft_consensus.cc:515] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.153543 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.153797 922 raft_consensus.cc:2468] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:21.154428 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:21.154606 780 raft_consensus.cc:3060] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.155916 2249 leader_election.cc:290] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.157462 780 raft_consensus.cc:2468] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:21.157896 998 leader_election.cc:304] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:21.158283 2249 raft_consensus.cc:2804] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.158448 2249 raft_consensus.cc:697] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:21.159442 2249 consensus_queue.cc:237] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.160400 1050 tablet_service.cc:1505] Processing CreateTablet for tablet cf71cd77855841b5a914a3056a130112 (DEFAULT_TABLE table=default.Table_c312fa246a2c4435b6a2c1fa67420ec0 [id=15d501a0104b4a1ab806d57b6cc32caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.160786 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf71cd77855841b5a914a3056a130112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.160703 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec54e736a3e64dd7a839252c440f778c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:21.160881 922 raft_consensus.cc:3060] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.163645 922 raft_consensus.cc:2468] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:21.166893 1177 tablet_bootstrap.cc:654] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.167729 1183 consensus_queue.cc:260] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.169430 1183 ts_tablet_manager.cc:1434] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20251029 08:52:21.169709 1183 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.170313 1179 ts_tablet_manager.cc:1929] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.181349 1179 log.cc:1199] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e4f80728a7a04583bada485707beb806
I20251029 08:52:21.181952 1179 ts_tablet_manager.cc:1950] T e4f80728a7a04583bada485707beb806 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.183368 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e4f80728a7a04583bada485707beb806 (table default.Table_18a7eae91e5b4ab595c36f630248860f [id=99ae7a11d37346e18a01a330d7fa42b1]) successfully deleted
I20251029 08:52:21.183912 1179 tablet_replica.cc:333] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.184136 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_919fdbb0653a4eb39f8cec21d90854c9" } new_table_name: "default.Table_5e2a0f712be74b21b5ca4007fb38bf9d" modify_external_catalogs: true
I20251029 08:52:21.186702 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a58b4ac66df64b0db81210ce141ece47" } new_table_name: "default.Table_7fd53f3b59d14ad39715bc62e513ab79" modify_external_catalogs: true
I20251029 08:52:21.187390 1179 raft_consensus.cc:2243] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.187618 1179 raft_consensus.cc:2272] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.185738 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_78e22440874d4598ac910219a652f3ed" } new_table_name: "default.Table_fe2750c35f884a3eb6ecce08b9a7aa7c" modify_external_catalogs: true
I20251029 08:52:21.188364 1179 ts_tablet_manager.cc:1916] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.192829 1177 tablet_bootstrap.cc:492] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.192994 1177 ts_tablet_manager.cc:1403] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20251029 08:52:21.197398 887 tablet_service.cc:1505] Processing CreateTablet for tablet 391bcab9c9584371afc26401edec8203 (DEFAULT_TABLE table=default.Table_ea4a80c608ad451aacff1c9c84ae64e1 [id=c821b6d0710f42cfb1faac91629455c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.199987 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_375cce62abe84482b107057a691c49f9" } new_table_name: "default.Table_7c44114212a547b4b4082d590f14e263" modify_external_catalogs: true
I20251029 08:52:21.200126 748 tablet_service.cc:1505] Processing CreateTablet for tablet cf71cd77855841b5a914a3056a130112 (DEFAULT_TABLE table=default.Table_c312fa246a2c4435b6a2c1fa67420ec0 [id=15d501a0104b4a1ab806d57b6cc32caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.200474 1179 ts_tablet_manager.cc:1929] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.200546 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf71cd77855841b5a914a3056a130112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.200575 1179 log.cc:1199] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/fd25eeb89e574297a92d8e978a79d768
I20251029 08:52:21.202502 2269 tablet.cc:1722] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:21.212281 1183 tablet_bootstrap.cc:654] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.214262 1179 ts_tablet_manager.cc:1950] T fd25eeb89e574297a92d8e978a79d768 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.215103 589 catalog_manager.cc:5649] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:21.217449 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66d11c1d0bd449aeb31f85ed681c6051" } modify_external_catalogs: true
I20251029 08:52:21.217644 2165 raft_consensus.cc:493] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.217677 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66d11c1d0bd449aeb31f85ed681c6051" } modify_external_catalogs: true
I20251029 08:52:21.217787 2165 raft_consensus.cc:515] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.218459 2165 leader_election.cc:290] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:21.219022 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c" } modify_external_catalogs: true
I20251029 08:52:21.219177 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c" } modify_external_catalogs: true
I20251029 08:52:21.220389 2165 raft_consensus.cc:493] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.220438 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c1a8a1e4cc14f1aa6e4f91174f45127" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.220515 2165 raft_consensus.cc:515] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.221053 2165 leader_election.cc:290] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.221871 749 tablet_service.cc:1505] Processing CreateTablet for tablet 391bcab9c9584371afc26401edec8203 (DEFAULT_TABLE table=default.Table_ea4a80c608ad451aacff1c9c84ae64e1 [id=c821b6d0710f42cfb1faac91629455c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.222290 753 tablet_service.cc:1505] Processing CreateTablet for tablet 55cfebce48b5454487a974ef9c98b1ee (DEFAULT_TABLE table=default.Table_6cc56e5870cf4447ab15779b3025731b [id=120fc62dca0b48a88b42df24443e3aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.222738 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cfebce48b5454487a974ef9c98b1ee. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.222815 720 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.223655 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 391bcab9c9584371afc26401edec8203 (DEFAULT_TABLE table=default.Table_ea4a80c608ad451aacff1c9c84ae64e1 [id=c821b6d0710f42cfb1faac91629455c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.223904 1179 tablet_replica.cc:333] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.224118 1179 raft_consensus.cc:2243] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.224251 1179 raft_consensus.cc:2272] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.224234 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet fd25eeb89e574297a92d8e978a79d768 (table default.Table_390ea1069fb74e8fa400c7c309ec231e [id=7ec37ecb910f4aeea495883f901b8f9b]) successfully deleted
I20251029 08:52:21.224534 745 tablet_service.cc:1505] Processing CreateTablet for tablet 165d46a006c445b7bf500de4f0a8ba70 (DEFAULT_TABLE table=default.Table_3421bab54d2445d09e0be9c5e15ac6da [id=f381b5dbf9b34d23930f73cdbca33689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.224768 1179 ts_tablet_manager.cc:1916] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.224864 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d46a006c445b7bf500de4f0a8ba70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.225245 1176 tablet_replica.cc:333] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:21.225226 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 55cfebce48b5454487a974ef9c98b1ee (DEFAULT_TABLE table=default.Table_6cc56e5870cf4447ab15779b3025731b [id=120fc62dca0b48a88b42df24443e3aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.225415 1176 raft_consensus.cc:2243] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.224256 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 165d46a006c445b7bf500de4f0a8ba70 (DEFAULT_TABLE table=default.Table_3421bab54d2445d09e0be9c5e15ac6da [id=f381b5dbf9b34d23930f73cdbca33689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.225538 1176 raft_consensus.cc:2272] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.226259 1176 ts_tablet_manager.cc:1916] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.226315 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cfebce48b5454487a974ef9c98b1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.228303 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d46a006c445b7bf500de4f0a8ba70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.229367 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391bcab9c9584371afc26401edec8203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.219627 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c1a8a1e4cc14f1aa6e4f91174f45127" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.229488 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0ef904c2fb4dfeadbf56ae659c4e20" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.229993 718 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.230096 718 leader_election.cc:304] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:21.230243 718 raft_consensus.cc:2686] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:21.230690 720 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.231635 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391bcab9c9584371afc26401edec8203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.238565 902 tablet_service.cc:1505] Processing CreateTablet for tablet 165d46a006c445b7bf500de4f0a8ba70 (DEFAULT_TABLE table=default.Table_3421bab54d2445d09e0be9c5e15ac6da [id=f381b5dbf9b34d23930f73cdbca33689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.239171 895 tablet_service.cc:1505] Processing CreateTablet for tablet 55cfebce48b5454487a974ef9c98b1ee (DEFAULT_TABLE table=default.Table_6cc56e5870cf4447ab15779b3025731b [id=120fc62dca0b48a88b42df24443e3aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.239959 1179 ts_tablet_manager.cc:1929] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.240105 1179 log.cc:1199] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d30183814f5e4f2ab9d71fe493eb801c
I20251029 08:52:21.241092 1179 ts_tablet_manager.cc:1950] T d30183814f5e4f2ab9d71fe493eb801c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.242138 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d30183814f5e4f2ab9d71fe493eb801c (table default.Table_753225d529ba4e54b0b04b6388919367 [id=159f32510e0843f5a3011cee6cc7c899]) successfully deleted
I20251029 08:52:21.242465 1179 tablet_replica.cc:333] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.242653 1179 raft_consensus.cc:2243] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.242796 1179 raft_consensus.cc:2272] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.243429 1179 ts_tablet_manager.cc:1916] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.246732 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5018010ef9af4f16b6388715c7f9fd03" } modify_external_catalogs: true
I20251029 08:52:21.246943 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5018010ef9af4f16b6388715c7f9fd03" } modify_external_catalogs: true
I20251029 08:52:21.222839 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0ef904c2fb4dfeadbf56ae659c4e20" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.248806 718 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.248950 718 leader_election.cc:304] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.249341 2165 raft_consensus.cc:2749] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.254287 1068 raft_consensus.cc:3060] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.255110 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391bcab9c9584371afc26401edec8203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.265610 2269 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:21.270912 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3e8df650164e496298534433c953840e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.272503 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8df650164e496298534433c953840e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.285090 1178 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.285266 1178 ts_tablet_manager.cc:1403] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.214s user 0.021s sys 0.057s
I20251029 08:52:21.285588 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3e111656804b44da9e8317170d1f3605" } modify_external_catalogs: true
I20251029 08:52:21.285913 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3e111656804b44da9e8317170d1f3605" } modify_external_catalogs: true
I20251029 08:52:21.286207 1178 raft_consensus.cc:359] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.286446 1178 raft_consensus.cc:385] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.286506 1178 raft_consensus.cc:740] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.286779 1178 consensus_queue.cc:260] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.287380 1178 ts_tablet_manager.cc:1434] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:21.291478 1178 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.292295 2165 raft_consensus.cc:493] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.292444 2165 raft_consensus.cc:515] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.293073 2165 leader_election.cc:290] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.292944 2284 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:21.293949 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.294189 1070 raft_consensus.cc:2468] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:21.294589 720 leader_election.cc:304] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:21.294891 2165 raft_consensus.cc:2804] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.294960 2165 raft_consensus.cc:493] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.295017 2165 raft_consensus.cc:3060] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.297585 1178 tablet_bootstrap.cc:654] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.297802 2165 raft_consensus.cc:515] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.298873 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:21.298887 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.299037 1066 raft_consensus.cc:3060] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:21.299329 718 leader_election.cc:343] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.301719 1066 raft_consensus.cc:2468] T 772de9216f2647f8ae52f594df272427 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:21.302904 720 leader_election.cc:304] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:21.305317 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5bc2f2f5d6e418999fa40a6fadbbc71 has no consensus state
I20251029 08:52:21.306816 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:21.307281 718 leader_election.cc:343] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.307601 2165 leader_election.cc:290] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.308070 2289 raft_consensus.cc:2804] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.312723 892 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd (DEFAULT_TABLE table=default.Table_8364a95e36094945a4d55d737dc1e117 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.313123 891 tablet_service.cc:1505] Processing CreateTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 (DEFAULT_TABLE table=default.Table_bf9b59ca50fd463eb0768ec552e1244a [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.313489 889 tablet_service.cc:1505] Processing CreateTablet for tablet 295ab2a10a5648abb59bc045c7dbe165 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.313979 896 tablet_service.cc:1505] Processing CreateTablet for tablet adc375f7a4104f899a05dba1be370f45 (DEFAULT_TABLE table=default.Table_45fb9d22c4e64c48b5c2b1312ca0b72e [id=4d0f944093794537bb81d890c5c4fd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.314474 890 tablet_service.cc:1505] Processing CreateTablet for tablet ef67d5cbc902492fb1252229b4b7f28f (DEFAULT_TABLE table=default.Table_50bc8d05fa8346ebaaa0c1a6a198d929 [id=7cf1bcfd0d684961b04aa470fb800054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.317808 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a513e19699c24ff58a3d6a6fcae4c96d" } modify_external_catalogs: true
I20251029 08:52:21.318838 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a513e19699c24ff58a3d6a6fcae4c96d" } modify_external_catalogs: true
I20251029 08:52:21.323570 1176 ts_tablet_manager.cc:1929] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:21.324334 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af913080bfe44a86a76704ffa32cd9e1 consensus state has no leader
I20251029 08:52:21.330538 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c204b45d751646c188724bd56add16c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.331143 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c204b45d751646c188724bd56add16c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.336807 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd (DEFAULT_TABLE table=default.Table_8364a95e36094945a4d55d737dc1e117 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.337218 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.338105 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cea6d85081f246788ba29bb4a9c9a4e8 consensus state has no leader
I20251029 08:52:21.338255 1045 tablet_service.cc:1505] Processing CreateTablet for tablet adc375f7a4104f899a05dba1be370f45 (DEFAULT_TABLE table=default.Table_45fb9d22c4e64c48b5c2b1312ca0b72e [id=4d0f944093794537bb81d890c5c4fd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.338564 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc375f7a4104f899a05dba1be370f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.339751 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bfb8a865be784f65ba33fe86cd9ee27e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.340340 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfb8a865be784f65ba33fe86cd9ee27e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.340378 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 (DEFAULT_TABLE table=default.Table_bf9b59ca50fd463eb0768ec552e1244a [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.340713 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5498826a9afd4fd5a6b88b0a465fff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.344308 1974 ts_tablet_manager.cc:1929] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.344269 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 295ab2a10a5648abb59bc045c7dbe165 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.344420 1974 log.cc:1199] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ea13e9ea2c11479ea86e3fd7f2a9e59c
I20251029 08:52:21.344606 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 295ab2a10a5648abb59bc045c7dbe165. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.349305 1045 tablet_service.cc:1505] Processing CreateTablet for tablet ef67d5cbc902492fb1252229b4b7f28f (DEFAULT_TABLE table=default.Table_50bc8d05fa8346ebaaa0c1a6a198d929 [id=7cf1bcfd0d684961b04aa470fb800054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.349918 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef67d5cbc902492fb1252229b4b7f28f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.356446 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00829c7fa0fe4e379a1a3044550e5941 consensus state has no leader
I20251029 08:52:21.358636 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0336e0b703a5476bbae10859daeee070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.359300 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0336e0b703a5476bbae10859daeee070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.359229 1177 raft_consensus.cc:359] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.364189 1177 raft_consensus.cc:385] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.364240 1177 raft_consensus.cc:740] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.364470 1177 consensus_queue.cc:260] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.365077 1177 ts_tablet_manager.cc:1434] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.172s user 0.002s sys 0.000s
I20251029 08:52:21.371583 1179 ts_tablet_manager.cc:1929] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.373932 1179 log.cc:1199] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/840ff254244f43a18679232279e02d1f
I20251029 08:52:21.374408 1179 ts_tablet_manager.cc:1950] T 840ff254244f43a18679232279e02d1f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.344736 1974 ts_tablet_manager.cc:1950] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.376971 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 840ff254244f43a18679232279e02d1f (table default.Table_82ec1f06f21147c991f019803452df8c [id=383565dcefdd479288ddfc261cbced6f]) successfully deleted
I20251029 08:52:21.378641 1179 tablet_replica.cc:333] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.378847 1179 raft_consensus.cc:2243] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.384375 1179 raft_consensus.cc:2272] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.384943 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d46a006c445b7bf500de4f0a8ba70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.385149 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef67d5cbc902492fb1252229b4b7f28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.385326 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc375f7a4104f899a05dba1be370f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.385497 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cfebce48b5454487a974ef9c98b1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.385726 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 295ab2a10a5648abb59bc045c7dbe165. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.385912 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.386094 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5498826a9afd4fd5a6b88b0a465fff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.400547 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]). Expected version 1 got 0
I20251029 08:52:21.401593 1177 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.402808 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:21.403604 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1b99bdda9ee74a1187811dccc1daf159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.405078 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b99bdda9ee74a1187811dccc1daf159 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.407181 1176 log.cc:1199] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:21.407567 1177 tablet_bootstrap.cc:654] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.410107 1176 ts_tablet_manager.cc:1950] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.418445 1183 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.418608 1183 ts_tablet_manager.cc:1403] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.249s user 0.002s sys 0.002s
I20251029 08:52:21.419389 1183 raft_consensus.cc:359] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.419554 1183 raft_consensus.cc:385] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.419605 1183 raft_consensus.cc:740] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.419791 1183 consensus_queue.cc:260] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.420631 1183 ts_tablet_manager.cc:1434] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:21.420965 1183 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.422932 1183 tablet_bootstrap.cc:654] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.424333 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (table default.Table_13d81683d3304abba1837078e6685abd [id=994af8859eb748c4a3ffee07d64d1e33]) successfully deleted
W20251029 08:52:21.425045 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fcaa3fa779824b38887ae001623ba67b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcaa3fa779824b38887ae001623ba67b already in progress: creating tablet
W20251029 08:52:21.428331 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 90360621ce124475af2f12060471f63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90360621ce124475af2f12060471f63c already in progress: creating tablet
I20251029 08:52:21.328233 2289 raft_consensus.cc:697] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:21.428792 2289 consensus_queue.cc:237] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.330842 2290 raft_consensus.cc:493] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.430292 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:21.430254 2290 raft_consensus.cc:515] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.331717 1178 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.336458 749 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd (DEFAULT_TABLE table=default.Table_8364a95e36094945a4d55d737dc1e117 [id=cf151f9ad5974db78cd578b1edfd034b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.430688 1178 ts_tablet_manager.cc:1403] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20251029 08:52:21.430858 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f048f1ae4d70aa1b88116a57e2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.430995 2290 leader_election.cc:290] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.432102 1178 raft_consensus.cc:359] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.432322 1178 raft_consensus.cc:385] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.432425 1178 raft_consensus.cc:740] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.432714 1178 consensus_queue.cc:260] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.433275 1178 ts_tablet_manager.cc:1434] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:21.433645 1178 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.337042 745 tablet_service.cc:1505] Processing CreateTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 (DEFAULT_TABLE table=default.Table_bf9b59ca50fd463eb0768ec552e1244a [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.435722 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5498826a9afd4fd5a6b88b0a465fff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.435995 1178 tablet_bootstrap.cc:654] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:21.438364 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 31bed64a27164baa8177b87b54b5947b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31bed64a27164baa8177b87b54b5947b already in progress: creating tablet
I20251029 08:52:21.438521 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.439039 718 leader_election.cc:343] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:21.439539 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cac7c7cf4679477bab723eec86c97ee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7c7cf4679477bab723eec86c97ee9 already in progress: creating tablet
W20251029 08:52:21.439901 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 34729c3f97a44713a352bde297a80027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34729c3f97a44713a352bde297a80027 already in progress: creating tablet
W20251029 08:52:21.440163 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a08630cbc401467a9dde1ce1e198cef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a08630cbc401467a9dde1ce1e198cef7 already in progress: creating tablet
I20251029 08:52:21.337541 753 tablet_service.cc:1505] Processing CreateTablet for tablet 295ab2a10a5648abb59bc045c7dbe165 (DEFAULT_TABLE table=default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.338892 757 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.440284 2165 raft_consensus.cc:493] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.344401 748 tablet_service.cc:1505] Processing CreateTablet for tablet adc375f7a4104f899a05dba1be370f45 (DEFAULT_TABLE table=default.Table_45fb9d22c4e64c48b5c2b1312ca0b72e [id=4d0f944093794537bb81d890c5c4fd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.440404 2165 raft_consensus.cc:515] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.440557 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 295ab2a10a5648abb59bc045c7dbe165. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.440984 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc375f7a4104f899a05dba1be370f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:21.441541 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.441987 718 leader_election.cc:343] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.440995 2165 leader_election.cc:290] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.346853 2294 tablet.cc:1722] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 08:52:21.444377 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cc7e3f702f9a407abf4fe7c7be513c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc7e3f702f9a407abf4fe7c7be513c67 already in progress: creating tablet
I20251029 08:52:21.444413 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.444772 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.444880 720 leader_election.cc:343] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.378293 758 tablet_service.cc:1505] Processing CreateTablet for tablet ef67d5cbc902492fb1252229b4b7f28f (DEFAULT_TABLE table=default.Table_50bc8d05fa8346ebaaa0c1a6a198d929 [id=7cf1bcfd0d684961b04aa470fb800054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:21.444969 720 leader_election.cc:304] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.445273 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef67d5cbc902492fb1252229b4b7f28f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:21.448726 720 leader_election.cc:343] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.448843 720 leader_election.cc:304] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.449267 2292 raft_consensus.cc:2749] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.449468 2292 raft_consensus.cc:2749] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.453598 1176 tablet_replica.cc:333] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:21.453675 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fcaa3fa779824b38887ae001623ba67b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcaa3fa779824b38887ae001623ba67b already in progress: creating tablet
I20251029 08:52:21.453892 1179 ts_tablet_manager.cc:1916] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.454205 1176 raft_consensus.cc:2243] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.454460 1176 raft_consensus.cc:2272] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.458024 2290 raft_consensus.cc:493] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.458290 2290 raft_consensus.cc:515] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.459944 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.460479 720 leader_election.cc:343] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.462266 2290 leader_election.cc:290] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.463409 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.463949 718 leader_election.cc:343] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.464073 718 leader_election.cc:304] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.464571 2290 raft_consensus.cc:2749] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.466603 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet afa2bb1075c4418eb10f4493f5e73c1b
I20251029 08:52:21.470320 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 29adc6d53af5419dbda1d41d75d37e71
I20251029 08:52:21.471607 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d26ff78aa5f94ae49a197daf2be00aa5
I20251029 08:52:21.473570 902 tablet_service.cc:1552] Processing DeleteTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.474179 1176 ts_tablet_manager.cc:1916] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:21.474228 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d26ff78aa5f94ae49a197daf2be00aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26ff78aa5f94ae49a197daf2be00aa5 already in progress: creating tablet
I20251029 08:52:21.474766 891 tablet_service.cc:1552] Processing DeleteTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:21.475364 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet afa2bb1075c4418eb10f4493f5e73c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa2bb1075c4418eb10f4493f5e73c1b already in progress: creating tablet
I20251029 08:52:21.475729 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.476752 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.477311 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.477304 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 29adc6d53af5419dbda1d41d75d37e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.477945 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.478389 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.478737 1974 tablet_replica.cc:333] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.478854 758 tablet_service.cc:1552] Processing DeleteTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.478926 1974 raft_consensus.cc:2243] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.479087 1974 raft_consensus.cc:2272] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.479631 1974 ts_tablet_manager.cc:1916] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.480096 1178 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.480260 1178 ts_tablet_manager.cc:1403] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.001s
I20251029 08:52:21.480063 597 catalog_manager.cc:5649] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:21.481355 1178 raft_consensus.cc:359] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.481645 1178 raft_consensus.cc:385] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.481827 1178 raft_consensus.cc:740] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.483858 1178 consensus_queue.cc:260] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.484926 1178 ts_tablet_manager.cc:1434] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:21.490263 1178 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.491786 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 13afd85e234643e08957e7a7f7fd7e22 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:21.497644 1178 tablet_bootstrap.cc:654] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.499459 2249 raft_consensus.cc:493] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.499605 2249 raft_consensus.cc:515] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.500061 1183 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.500211 1183 ts_tablet_manager.cc:1403] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.001s
I20251029 08:52:21.500442 2249 leader_election.cc:290] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.501089 1183 raft_consensus.cc:359] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.501236 1183 raft_consensus.cc:385] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.501291 1183 raft_consensus.cc:740] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:21.501637 996 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 14123bfd9c3a4a6aba8cacd92e0ac0bc
W20251029 08:52:21.502499 998 leader_election.cc:343] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:21.502592 998 leader_election.cc:304] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.503007 2249 raft_consensus.cc:2749] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.503356 1183 consensus_queue.cc:260] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.504289 1183 ts_tablet_manager.cc:1434] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:21.506067 2290 raft_consensus.cc:493] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.506500 2290 raft_consensus.cc:515] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.510284 2290 leader_election.cc:290] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:21.510562 1183 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.510895 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:21.511310 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 898a4f58306b453e9abca2a73082bb96 (table default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa]). Expected version 1 got 0
I20251029 08:52:21.511559 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet cc57cd5a8d3c42ba90f2011084e8b4bd (table default.Table_7e364318a48e4b8eb94a67fcd126a67b [id=5dc0a7f470cb432dbd918d84b96dd58b]). Expected version 2 got 0
I20251029 08:52:21.511883 594 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 19c515944ecf4228a7cca63cfd3565d8 (table default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46]). Expected version 1 got 0
I20251029 08:52:21.512986 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cdcc9a441c6a4d82a58262f4fdb3f00d
I20251029 08:52:21.514694 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afa2bb1075c4418eb10f4493f5e73c1b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.515175 720 leader_election.cc:343] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.515810 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.516324 758 tablet_service.cc:1552] Processing DeleteTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.517263 889 tablet_service.cc:1552] Processing DeleteTablet for tablet cdcc9a441c6a4d82a58262f4fdb3f00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.517531 1183 tablet_bootstrap.cc:654] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.522636 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cafb2beec81f464486fab880551e8b66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.523413 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cafb2beec81f464486fab880551e8b66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.526422 2269 tablet.cc:1722] T 19c515944ecf4228a7cca63cfd3565d8 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:21.528128 2284 tablet.cc:1722] T 19c515944ecf4228a7cca63cfd3565d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:21.531957 2290 raft_consensus.cc:493] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.532406 2290 raft_consensus.cc:515] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.534974 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.535953 720 leader_election.cc:343] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.536374 591 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet cbf00fc9ac114f7e9fff87d59a93100f
I20251029 08:52:21.537227 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.537367 2290 leader_election.cc:290] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.537931 1179 ts_tablet_manager.cc:1929] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.538044 1179 log.cc:1199] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/8991ea78321545be85d5c480841e4751
I20251029 08:52:21.538389 1179 ts_tablet_manager.cc:1950] T 8991ea78321545be85d5c480841e4751 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:21.537699 718 leader_election.cc:343] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.539496 718 leader_election.cc:304] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.534956 1183 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.539749 1183 ts_tablet_manager.cc:1403] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20251029 08:52:21.539861 2165 raft_consensus.cc:2749] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.540046 2165 raft_consensus.cc:493] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.540184 2165 raft_consensus.cc:515] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.540688 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 8991ea78321545be85d5c480841e4751 (table default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]) successfully deleted
I20251029 08:52:21.540802 2165 leader_election.cc:290] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.541430 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.541618 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.542022 718 leader_election.cc:343] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:21.542271 720 leader_election.cc:343] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.542389 720 leader_election.cc:304] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.542725 2289 raft_consensus.cc:2749] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.509814 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afa2bb1075c4418eb10f4493f5e73c1b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.543339 1183 raft_consensus.cc:359] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:21.543474 718 leader_election.cc:343] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.543471 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_554113ad7eea4ccfa7b3b4af7f145923" } new_table_name: "default.Table_6472d3c59db1490eb91b81a60372f0d4" modify_external_catalogs: true
I20251029 08:52:21.543593 718 leader_election.cc:304] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.543931 2290 raft_consensus.cc:2749] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.544196 1183 raft_consensus.cc:385] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.544267 1183 raft_consensus.cc:740] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.544468 1183 consensus_queue.cc:260] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.546339 1183 ts_tablet_manager.cc:1434] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:21.547180 1183 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:21.548982 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a712185fe4aa4f8faf1ce2df229c11ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a712185fe4aa4f8faf1ce2df229c11ee already in progress: creating tablet
I20251029 08:52:21.549777 1179 tablet_replica.cc:333] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.550312 1179 raft_consensus.cc:2243] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:21.550647 1177 tablet_bootstrap.cc:492] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.550832 1177 ts_tablet_manager.cc:1403] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.007s
I20251029 08:52:21.551061 1179 pending_rounds.cc:70] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:21.551162 1179 pending_rounds.cc:77] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037646552489984 op_type: NO_OP noop_request { }
I20251029 08:52:21.551853 1179 raft_consensus.cc:2889] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:21.552048 1179 raft_consensus.cc:2272] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.552141 1177 raft_consensus.cc:359] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.552376 1177 raft_consensus.cc:385] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.552426 1177 raft_consensus.cc:740] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.552680 1183 tablet_bootstrap.cc:654] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.552662 1177 consensus_queue.cc:260] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.553153 1177 ts_tablet_manager.cc:1434] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:21.553462 1177 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.557439 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c3fd597fc0584f51b48998d2c8794591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.557695 1178 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.557861 1178 ts_tablet_manager.cc:1403] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
W20251029 08:52:21.558136 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3fd597fc0584f51b48998d2c8794591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.558714 1177 tablet_bootstrap.cc:654] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.558704 1178 raft_consensus.cc:359] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.558894 1178 raft_consensus.cc:385] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.558962 1178 raft_consensus.cc:740] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.556460 1974 ts_tablet_manager.cc:1929] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:21.559059 1974 log.cc:1199] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/29adc6d53af5419dbda1d41d75d37e71
I20251029 08:52:21.559244 1178 consensus_queue.cc:260] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.559758 1178 ts_tablet_manager.cc:1434] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:21.560873 1179 ts_tablet_manager.cc:1916] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.567232 1178 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.568356 1974 ts_tablet_manager.cc:1950] T 29adc6d53af5419dbda1d41d75d37e71 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.569000 1974 tablet_replica.cc:333] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.573249 1178 tablet_bootstrap.cc:654] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.573266 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
I20251029 08:52:21.574491 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2de7c30e5ad643dc9cbe84972d3f3f54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.574765 1974 raft_consensus.cc:2243] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:21.575091 1974 raft_consensus.cc:2272] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.575146 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2de7c30e5ad643dc9cbe84972d3f3f54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.576834 2249 raft_consensus.cc:493] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.576982 2249 raft_consensus.cc:515] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.577600 2249 leader_election.cc:290] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:21.579172 996 leader_election.cc:343] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 449ffc678b874871a672c3059235de6a
W20251029 08:52:21.579559 998 leader_election.cc:343] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 449ffc678b874871a672c3059235de6a
I20251029 08:52:21.579663 998 leader_election.cc:304] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.580094 2249 raft_consensus.cc:2749] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.581934 1183 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.582113 1183 ts_tablet_manager.cc:1403] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20251029 08:52:21.582453 1974 ts_tablet_manager.cc:1916] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.582957 1183 raft_consensus.cc:359] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.583154 1183 raft_consensus.cc:385] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.583226 1183 raft_consensus.cc:740] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.583446 1183 consensus_queue.cc:260] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.584846 1177 tablet_bootstrap.cc:492] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.584995 1177 ts_tablet_manager.cc:1403] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251029 08:52:21.585758 1177 raft_consensus.cc:359] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.585929 1177 raft_consensus.cc:385] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.585819 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_df009bebc8ca4185b078a085e10c3499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.585990 1177 raft_consensus.cc:740] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.586278 1183 ts_tablet_manager.cc:1434] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 08:52:21.586208 1177 consensus_queue.cc:260] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:21.586462 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df009bebc8ca4185b078a085e10c3499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.586588 1183 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.594652 2294 tablet.cc:1722] T 19c515944ecf4228a7cca63cfd3565d8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:21.595463 1183 tablet_bootstrap.cc:654] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.596058 1177 ts_tablet_manager.cc:1434] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251029 08:52:21.598196 589 master_service.cc:954] ReplaceTablet: received request to replace tablet a4ae5978b71443a48a88bc69128b187d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:21.599624 1177 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.594686 1179 ts_tablet_manager.cc:1929] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:21.600749 1179 log.cc:1199] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ea13e9ea2c11479ea86e3fd7f2a9e59c
I20251029 08:52:21.601192 1179 ts_tablet_manager.cc:1950] T ea13e9ea2c11479ea86e3fd7f2a9e59c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.602442 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ea13e9ea2c11479ea86e3fd7f2a9e59c (table default.Table_13d81683d3304abba1837078e6685abd [id=994af8859eb748c4a3ffee07d64d1e33]) successfully deleted
I20251029 08:52:21.602926 1177 tablet_bootstrap.cc:654] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.608562 1183 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.608721 1183 ts_tablet_manager.cc:1403] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20251029 08:52:21.614252 1176 ts_tablet_manager.cc:1929] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.618095 590 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_d707b992dd69414897e525d86fb4b002 [id=a23a597c8fd343769b517507845104ff] alter complete (version 2)
I20251029 08:52:21.622577 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 084a659932ec4996a6d6b160d67efdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.624064 1183 raft_consensus.cc:359] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.624261 1183 raft_consensus.cc:385] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.624320 1183 raft_consensus.cc:740] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.624640 1179 tablet_replica.cc:333] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.624878 1179 raft_consensus.cc:2243] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.625041 1177 tablet_bootstrap.cc:492] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.625087 1179 raft_consensus.cc:2272] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.625165 1177 ts_tablet_manager.cc:1403] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.002s
I20251029 08:52:21.625667 1179 ts_tablet_manager.cc:1916] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.625886 1177 raft_consensus.cc:359] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.626087 1177 raft_consensus.cc:385] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.626129 1177 raft_consensus.cc:740] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.626325 1177 consensus_queue.cc:260] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.628090 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d378e85a66bb4e40b69d49ec1cf86a51" } new_table_name: "default.Table_4e34d668a8db45a9b820969bb5939129" modify_external_catalogs: true
I20251029 08:52:21.629634 1183 consensus_queue.cc:260] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.629823 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_65e541ea2b244bf096d9362726620996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.630366 1177 ts_tablet_manager.cc:1434] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251029 08:52:21.630496 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65e541ea2b244bf096d9362726620996 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.632910 1183 ts_tablet_manager.cc:1434] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251029 08:52:21.633226 1183 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.634052 1177 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.635797 1177 tablet_bootstrap.cc:654] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.637881 1176 log.cc:1199] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:21.641218 1183 tablet_bootstrap.cc:654] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.642535 1177 tablet_bootstrap.cc:492] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.642668 1177 ts_tablet_manager.cc:1403] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251029 08:52:21.643349 1177 raft_consensus.cc:359] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.643520 1177 raft_consensus.cc:385] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.643599 1177 raft_consensus.cc:740] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.643910 1177 consensus_queue.cc:260] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.648254 1177 ts_tablet_manager.cc:1434] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:21.652417 1177 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.654138 1177 tablet_bootstrap.cc:654] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.657078 1177 tablet_bootstrap.cc:492] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.657224 1177 ts_tablet_manager.cc:1403] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251029 08:52:21.657991 1177 raft_consensus.cc:359] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:21.658121 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e2554e46fb44660893e95a0654306e9 has no consensus state
I20251029 08:52:21.658159 1177 raft_consensus.cc:385] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.658206 1177 raft_consensus.cc:740] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.658442 1177 consensus_queue.cc:260] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.662757 1177 ts_tablet_manager.cc:1434] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:21.664220 1178 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.664392 1178 ts_tablet_manager.cc:1403] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.007s
I20251029 08:52:21.665195 1178 raft_consensus.cc:359] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.665376 1178 raft_consensus.cc:385] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.665452 1178 raft_consensus.cc:740] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.665814 1178 consensus_queue.cc:260] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.667074 1177 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.667388 1178 ts_tablet_manager.cc:1434] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:21.667681 1178 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.670877 1178 tablet_bootstrap.cc:654] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.671123 1177 tablet_bootstrap.cc:654] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.677635 1177 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.677800 1177 ts_tablet_manager.cc:1403] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20251029 08:52:21.678529 1177 raft_consensus.cc:359] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.678712 1177 raft_consensus.cc:385] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.678761 1177 raft_consensus.cc:740] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.679070 1177 consensus_queue.cc:260] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.680828 1183 tablet_bootstrap.cc:492] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.680991 1183 ts_tablet_manager.cc:1403] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20251029 08:52:21.681346 2295 raft_consensus.cc:493] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.681471 2295 raft_consensus.cc:515] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.681586 1179 ts_tablet_manager.cc:1929] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.681730 1179 log.cc:1199] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/449ffc678b874871a672c3059235de6a
I20251029 08:52:21.682063 2295 leader_election.cc:290] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:21.682103 1179 ts_tablet_manager.cc:1950] T 449ffc678b874871a672c3059235de6a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.682189 1183 raft_consensus.cc:359] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.682358 1183 raft_consensus.cc:385] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.682420 1183 raft_consensus.cc:740] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.682644 1183 consensus_queue.cc:260] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.682754 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bda3e143699b41b6bba308c83bbb3e0d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.682900 1068 raft_consensus.cc:2468] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:21.683164 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 449ffc678b874871a672c3059235de6a (table default.Table_6f454ac2adfa4a4da06fa1180ca26c58 [id=4245017c2ec4498aa0f1200cf74040ff]) successfully deleted
I20251029 08:52:21.684235 1183 ts_tablet_manager.cc:1434] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251029 08:52:21.684291 861 leader_election.cc:343] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bda3e143699b41b6bba308c83bbb3e0d
I20251029 08:52:21.684494 861 leader_election.cc:304] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:21.686144 2199 raft_consensus.cc:2804] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.686219 2199 raft_consensus.cc:493] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.686273 2199 raft_consensus.cc:3060] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:21.687556 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 31bed64a27164baa8177b87b54b5947b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31bed64a27164baa8177b87b54b5947b already in progress: creating tablet
I20251029 08:52:21.688047 1179 tablet_replica.cc:333] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.688310 1179 raft_consensus.cc:2243] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:21.688601 1179 pending_rounds.cc:70] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:21.688663 1179 pending_rounds.cc:77] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037646916063232 op_type: NO_OP noop_request { }
I20251029 08:52:21.688787 1179 raft_consensus.cc:2889] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:21.688876 1179 raft_consensus.cc:2272] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.689157 2199 raft_consensus.cc:515] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.689796 2199 leader_election.cc:290] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:21.692965 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bda3e143699b41b6bba308c83bbb3e0d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:21.693152 1068 raft_consensus.cc:3060] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.694216 1179 ts_tablet_manager.cc:1916] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:21.696681 861 leader_election.cc:343] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bda3e143699b41b6bba308c83bbb3e0d
I20251029 08:52:21.697563 1068 raft_consensus.cc:2468] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:21.698104 861 leader_election.cc:304] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:21.698628 2295 raft_consensus.cc:2804] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.698769 2295 raft_consensus.cc:697] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:21.699045 2295 consensus_queue.cc:237] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.705387 1183 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.706957 596 master_service.cc:954] ReplaceTablet: received request to replace tablet a013257089774ba8b1f535e5f94ac1bb from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:21.707901 996 consensus_peers.cc:597] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29adc6d53af5419dbda1d41d75d37e71. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:21.714241 1183 tablet_bootstrap.cc:654] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.716884 2290 raft_consensus.cc:493] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.717056 2290 raft_consensus.cc:515] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.717741 2290 leader_election.cc:290] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.718627 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3955ff79faf42bb82f764aae79b85d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.719192 718 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.719663 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3955ff79faf42bb82f764aae79b85d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.720309 720 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.720451 720 leader_election.cc:304] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.720448 1179 ts_tablet_manager.cc:1929] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:21.720587 1179 log.cc:1199] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:21.720841 2290 raft_consensus.cc:2749] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.721009 1179 ts_tablet_manager.cc:1950] T 084a659932ec4996a6d6b160d67efdd0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.722294 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 084a659932ec4996a6d6b160d67efdd0 (table default.Table_c36b90fc430a4a89ba87469be66cb2a5 [id=d88876b1c5514d78892db64b612182ba]) successfully deleted
I20251029 08:52:21.725320 1178 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.725467 1178 ts_tablet_manager.cc:1403] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.008s
I20251029 08:52:21.726177 1178 raft_consensus.cc:359] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.726347 1178 raft_consensus.cc:385] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.726399 1178 raft_consensus.cc:740] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.726648 1178 consensus_queue.cc:260] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:21.728246 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fcaa3fa779824b38887ae001623ba67b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcaa3fa779824b38887ae001623ba67b already in progress: creating tablet
I20251029 08:52:21.729197 1178 ts_tablet_manager.cc:1434] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:21.729485 1178 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:21.730636 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 90360621ce124475af2f12060471f63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90360621ce124475af2f12060471f63c already in progress: creating tablet
I20251029 08:52:21.731205 1178 tablet_bootstrap.cc:654] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:21.731964 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cc7e3f702f9a407abf4fe7c7be513c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc7e3f702f9a407abf4fe7c7be513c67 already in progress: creating tablet
I20251029 08:52:21.733487 1177 ts_tablet_manager.cc:1434] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.056s user 0.000s sys 0.001s
I20251029 08:52:21.734184 1177 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.734683 1178 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.734818 1178 ts_tablet_manager.cc:1403] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251029 08:52:21.735572 1177 tablet_bootstrap.cc:654] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.735505 1178 raft_consensus.cc:359] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.735675 1178 raft_consensus.cc:385] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.735723 1178 raft_consensus.cc:740] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.735944 1178 consensus_queue.cc:260] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.736410 1176 ts_tablet_manager.cc:1950] T 627bc8ff99394fa2abcca15d62e5ea10 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.736881 1178 ts_tablet_manager.cc:1434] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:21.737511 1178 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:21.737592 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cac7c7cf4679477bab723eec86c97ee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7c7cf4679477bab723eec86c97ee9 already in progress: creating tablet
W20251029 08:52:21.738498 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 34729c3f97a44713a352bde297a80027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34729c3f97a44713a352bde297a80027 already in progress: creating tablet
W20251029 08:52:21.739701 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a08630cbc401467a9dde1ce1e198cef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a08630cbc401467a9dde1ce1e198cef7 already in progress: creating tablet
I20251029 08:52:21.739960 1178 tablet_bootstrap.cc:654] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.740617 1179 tablet_replica.cc:333] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.740872 1179 raft_consensus.cc:2243] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.741025 1179 raft_consensus.cc:2272] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.741652 1179 ts_tablet_manager.cc:1916] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.742208 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 627bc8ff99394fa2abcca15d62e5ea10 (table default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]) successfully deleted
I20251029 08:52:21.742648 1176 tablet_replica.cc:333] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:21.742857 1176 raft_consensus.cc:2243] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.743037 1176 raft_consensus.cc:2272] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.743604 1176 ts_tablet_manager.cc:1916] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.757198 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fe464f17db3547f393d347ee7c8643c2" } new_table_name: "default.Table_f85fb298cb3444939f91712dd2da4525" modify_external_catalogs: true
I20251029 08:52:21.758033 1176 ts_tablet_manager.cc:1929] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.758141 1176 log.cc:1199] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/ba201a54466240129d82a096f9316242
I20251029 08:52:21.758415 1176 ts_tablet_manager.cc:1950] T ba201a54466240129d82a096f9316242 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.759032 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5e52e5119f2d4324946d85f98d69f62b" } new_table_name: "default.Table_24dc6417d2284eb380c010cc2586067c" modify_external_catalogs: true
I20251029 08:52:21.759462 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet ba201a54466240129d82a096f9316242 (table default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]) successfully deleted
I20251029 08:52:21.761445 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_61da851751cb48c99d7c6dded2f21a43" } new_table_name: "default.Table_01aee9ace77c47d3ad4ef31880852a51" modify_external_catalogs: true
I20251029 08:52:21.762295 1974 ts_tablet_manager.cc:1929] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:21.762425 1974 log.cc:1199] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/cdcc9a441c6a4d82a58262f4fdb3f00d
I20251029 08:52:21.763695 1974 ts_tablet_manager.cc:1950] T cdcc9a441c6a4d82a58262f4fdb3f00d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.763813 1176 tablet_replica.cc:333] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:21.764052 1176 raft_consensus.cc:2243] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.764217 1176 raft_consensus.cc:2272] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.764694 1179 ts_tablet_manager.cc:1929] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.764722 1176 ts_tablet_manager.cc:1916] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.764820 1179 log.cc:1199] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/14123bfd9c3a4a6aba8cacd92e0ac0bc
I20251029 08:52:21.765159 1179 ts_tablet_manager.cc:1950] T 14123bfd9c3a4a6aba8cacd92e0ac0bc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.765506 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]) successfully deleted
I20251029 08:52:21.765880 1974 tablet_replica.cc:333] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.766104 1974 raft_consensus.cc:2243] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.766206 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 14123bfd9c3a4a6aba8cacd92e0ac0bc (table default.Table_4f1f60e2c98f47079dc3996b94e0d4c7 [id=56ee897e68cb4f89babe8102a39458b7]) successfully deleted
I20251029 08:52:21.766884 1974 raft_consensus.cc:2272] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.767405 1974 ts_tablet_manager.cc:1916] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.770700 1176 ts_tablet_manager.cc:1929] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.770807 1176 log.cc:1199] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:21.770874 1179 tablet_replica.cc:333] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.771059 1179 raft_consensus.cc:2243] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.771226 1179 raft_consensus.cc:2272] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.773264 1179 ts_tablet_manager.cc:1916] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.776542 1974 ts_tablet_manager.cc:1929] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.776716 1974 log.cc:1199] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/084a659932ec4996a6d6b160d67efdd0
I20251029 08:52:21.777041 1974 ts_tablet_manager.cc:1950] T 084a659932ec4996a6d6b160d67efdd0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.778028 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 084a659932ec4996a6d6b160d67efdd0 (table default.Table_c36b90fc430a4a89ba87469be66cb2a5 [id=d88876b1c5514d78892db64b612182ba]) successfully deleted
I20251029 08:52:21.779604 1179 ts_tablet_manager.cc:1929] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.779718 1179 log.cc:1199] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/afa2bb1075c4418eb10f4493f5e73c1b
I20251029 08:52:21.783358 1183 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.787745 1183 ts_tablet_manager.cc:1403] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.012s
I20251029 08:52:21.788558 1183 raft_consensus.cc:359] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.788741 1183 raft_consensus.cc:385] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.788805 1183 raft_consensus.cc:740] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.789055 1183 consensus_queue.cc:260] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.789108 594 master_service.cc:954] ReplaceTablet: received request to replace tablet dbf9e36c47be4509a998be8e7138475b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:21.790591 597 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_1c1e2303aee3434c959bd347637bae57 [id=92e293be9adf4ae9a20e0a42e5277e46] alter complete (version 1)
I20251029 08:52:21.791431 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_892ad3428acd46749ff80438c5b8f0d3" } modify_external_catalogs: true
I20251029 08:52:21.791591 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_892ad3428acd46749ff80438c5b8f0d3" } modify_external_catalogs: true
I20251029 08:52:21.791767 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.791780 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2723cdd513ee4e8ca6773b38f785b09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.792105 1974 tablet_replica.cc:333] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.792279 1974 raft_consensus.cc:2243] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.792423 1974 raft_consensus.cc:2272] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.793061 1183 ts_tablet_manager.cc:1434] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:21.793360 1183 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.793771 1974 ts_tablet_manager.cc:1916] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.794468 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c30d9ab9918b4baaa0a6d733432364c1" } modify_external_catalogs: true
I20251029 08:52:21.794648 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c30d9ab9918b4baaa0a6d733432364c1" } modify_external_catalogs: true
I20251029 08:52:21.795133 1183 tablet_bootstrap.cc:654] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.795889 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_54c3fb69aa194818946513e7b678f20a" } modify_external_catalogs: true
I20251029 08:52:21.796085 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_54c3fb69aa194818946513e7b678f20a" } modify_external_catalogs: true
I20251029 08:52:21.799201 1176 ts_tablet_manager.cc:1950] T aa4f97075b3941aaa7cda3f9413e867f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.790782 1177 tablet_bootstrap.cc:492] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:21.800180 1177 ts_tablet_manager.cc:1403] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20251029 08:52:21.800876 1176 tablet_replica.cc:333] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:21.800937 1177 raft_consensus.cc:359] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.801091 1177 raft_consensus.cc:385] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.801151 1177 raft_consensus.cc:740] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.801400 1177 consensus_queue.cc:260] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.802143 590 master_service.cc:954] ReplaceTablet: received request to replace tablet af38662420d14ac1a7cb5fd03fc1b9d8 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:21.802783 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f04c7f07ea40494aaf173ed78172c756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.803371 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f04c7f07ea40494aaf173ed78172c756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.780004 1179 ts_tablet_manager.cc:1950] T afa2bb1075c4418eb10f4493f5e73c1b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.804852 1177 ts_tablet_manager.cc:1434] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:21.805107 1177 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:21.805614 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet bda3e143699b41b6bba308c83bbb3e0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.806555 1177 tablet_bootstrap.cc:654] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.805982 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f40aafe7dc94851a67abd84d80d7d7b with delete_type TABLET_DATA_DELETED (replaced by tablet 15dae38bcced496e90ca8eec280a8317) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.806030 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.806370 1183 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.806960 1183 ts_tablet_manager.cc:1403] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20251029 08:52:21.807343 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.805522 1179 tablet_replica.cc:333] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.807560 749 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.808004 1183 raft_consensus.cc:359] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.808327 1183 raft_consensus.cc:385] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.808254 1179 raft_consensus.cc:2243] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:21.808661 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a4ae5978b71443a48a88bc69128b187d
I20251029 08:52:21.808861 1179 raft_consensus.cc:2272] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.809183 589 catalog_manager.cc:6719] ReplaceTablet: tablet a4ae5978b71443a48a88bc69128b187d deleted and replaced by tablet 8a8accf8f9eb4e15914287e49cc8180f
I20251029 08:52:21.809175 1974 ts_tablet_manager.cc:1929] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.809333 1974 log.cc:1199] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:21.809504 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d6e32e655391437ea03d14eade0cc35f" } modify_external_catalogs: true
I20251029 08:52:21.809602 1974 ts_tablet_manager.cc:1950] T 2f40aafe7dc94851a67abd84d80d7d7b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.809671 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d6e32e655391437ea03d14eade0cc35f" } modify_external_catalogs: true
I20251029 08:52:21.810040 1183 raft_consensus.cc:740] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.810268 1183 consensus_queue.cc:260] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.810364 891 tablet_service.cc:1552] Processing DeleteTablet for tablet a4ae5978b71443a48a88bc69128b187d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8accf8f9eb4e15914287e49cc8180f) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.810611 1179 ts_tablet_manager.cc:1916] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.813071 1974 tablet_replica.cc:333] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:21.813248 1974 raft_consensus.cc:2243] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.813403 1974 raft_consensus.cc:2272] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.813896 1183 ts_tablet_manager.cc:1434] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:21.814204 1183 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.814895 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 2f40aafe7dc94851a67abd84d80d7d7b (table default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]) successfully deleted
I20251029 08:52:21.816634 1974 ts_tablet_manager.cc:1916] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.816991 1178 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.817122 1178 ts_tablet_manager.cc:1403] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.004s
I20251029 08:52:21.817876 1178 raft_consensus.cc:359] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.818037 1178 raft_consensus.cc:385] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.818096 1178 raft_consensus.cc:740] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.818341 1178 consensus_queue.cc:260] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.821430 1178 ts_tablet_manager.cc:1434] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:21.821730 1178 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.822981 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet aa4f97075b3941aaa7cda3f9413e867f (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
W20251029 08:52:21.823280 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 because the tablet was not found. No further retry: Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
W20251029 08:52:21.823477 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 90360621ce124475af2f12060471f63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90360621ce124475af2f12060471f63c already in progress: creating tablet
W20251029 08:52:21.823660 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cc7e3f702f9a407abf4fe7c7be513c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc7e3f702f9a407abf4fe7c7be513c67 already in progress: creating tablet
W20251029 08:52:21.823839 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cac7c7cf4679477bab723eec86c97ee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7c7cf4679477bab723eec86c97ee9 already in progress: creating tablet
I20251029 08:52:21.824050 1178 tablet_bootstrap.cc:654] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:21.824083 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a08630cbc401467a9dde1ce1e198cef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a08630cbc401467a9dde1ce1e198cef7 already in progress: creating tablet
W20251029 08:52:21.824272 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 34729c3f97a44713a352bde297a80027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34729c3f97a44713a352bde297a80027 already in progress: creating tablet
I20251029 08:52:21.824466 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet afa2bb1075c4418eb10f4493f5e73c1b (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:21.826647 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet a4ae5978b71443a48a88bc69128b187d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8accf8f9eb4e15914287e49cc8180f) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.815342 758 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.827419 1183 tablet_bootstrap.cc:654] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.828148 758 tablet_service.cc:1552] Processing DeleteTablet for tablet a4ae5978b71443a48a88bc69128b187d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8accf8f9eb4e15914287e49cc8180f) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.828358 2199 raft_consensus.cc:493] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.828517 2199 raft_consensus.cc:515] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.829118 2199 leader_election.cc:290] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:21.828150 1176 raft_consensus.cc:2243] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.830093 1176 raft_consensus.cc:2272] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.833190 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72d0540040614513a4518ac983d14a94 consensus state has no leader
I20251029 08:52:21.833400 1176 ts_tablet_manager.cc:1916] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:21.835300 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19ab7e172a704db3aa4c32da35993003 has no consensus state
I20251029 08:52:21.835680 1974 ts_tablet_manager.cc:1929] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.835781 1974 log.cc:1199] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/2723cdd513ee4e8ca6773b38f785b09e
I20251029 08:52:21.839893 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351047c725f8488abbd074bd40790b48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.840107 1068 raft_consensus.cc:2468] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:21.840603 861 leader_election.cc:304] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:21.840981 861 leader_election.cc:343] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 351047c725f8488abbd074bd40790b48
I20251029 08:52:21.841732 2199 raft_consensus.cc:2804] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.841831 2199 raft_consensus.cc:493] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.841907 2199 raft_consensus.cc:3060] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.846573 1179 ts_tablet_manager.cc:1929] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:21.846685 1179 log.cc:1199] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/29adc6d53af5419dbda1d41d75d37e71
I20251029 08:52:21.847025 1179 ts_tablet_manager.cc:1950] T 29adc6d53af5419dbda1d41d75d37e71 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.847474 2199 raft_consensus.cc:515] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.847702 1176 ts_tablet_manager.cc:1929] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.847855 1176 log.cc:1199] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:21.848889 1178 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.849018 1178 ts_tablet_manager.cc:1403] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251029 08:52:21.848940 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351047c725f8488abbd074bd40790b48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:21.849140 1068 raft_consensus.cc:3060] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.849712 1178 raft_consensus.cc:359] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.850085 1178 raft_consensus.cc:385] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.850144 1178 raft_consensus.cc:740] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.850375 1178 consensus_queue.cc:260] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.851400 2295 raft_consensus.cc:493] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.851526 2295 raft_consensus.cc:515] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:21.851495 1974 ts_tablet_manager.cc:1950] T 2723cdd513ee4e8ca6773b38f785b09e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:21.852711 1068 raft_consensus.cc:2468] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:21.853153 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 2723cdd513ee4e8ca6773b38f785b09e (table default.Table_796d0f3f111e416aa8c95689e9d80a73 [id=ab706a4605ac4b01a31c0afad230e790]) successfully deleted
I20251029 08:52:21.853371 2199 leader_election.cc:290] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:21.853511 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a4ae5978b71443a48a88bc69128b187d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4ae5978b71443a48a88bc69128b187d already in progress: creating tablet
I20251029 08:52:21.853899 861 leader_election.cc:304] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:21.854156 861 leader_election.cc:343] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a3ba512bd4f04bd9aceb4cb210fe9689
I20251029 08:52:21.854225 1178 ts_tablet_manager.cc:1434] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:21.854323 1179 tablet_replica.cc:333] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:21.854883 861 leader_election.cc:343] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 351047c725f8488abbd074bd40790b48
I20251029 08:52:21.854955 1179 raft_consensus.cc:2243] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.855161 1178 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.855298 2199 raft_consensus.cc:2804] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:21.855944 1179 raft_consensus.cc:2272] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.856479 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82fdad2bc86541b4ae20690fae62fb42 consensus state has no leader
I20251029 08:52:21.857270 1179 ts_tablet_manager.cc:1916] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.858053 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2db77250e3ef489e8ab7dbcfcbb54385 (table default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81]). Expected version 2 got 0
I20251029 08:52:21.858361 2199 raft_consensus.cc:697] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:21.858412 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:21.858583 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet dbf9e36c47be4509a998be8e7138475b
I20251029 08:52:21.858639 2199 consensus_queue.cc:237] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.859066 594 catalog_manager.cc:6719] ReplaceTablet: tablet dbf9e36c47be4509a998be8e7138475b deleted and replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7
I20251029 08:52:21.859349 891 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf9e36c47be4509a998be8e7138475b with delete_type TABLET_DATA_DELETED (replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.859373 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c3fd597fc0584f51b48998d2c8794591" } modify_external_catalogs: true
I20251029 08:52:21.859572 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c3fd597fc0584f51b48998d2c8794591" } modify_external_catalogs: true
I20251029 08:52:21.860435 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_755a398e8bef4cbd9268a01dff6fd627"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:21.861024 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_755a398e8bef4cbd9268a01dff6fd627 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.861795 922 raft_consensus.cc:3060] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.862341 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_96c75b14349a481184cd42dd8b8ede6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.856479 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
W20251029 08:52:21.862826 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96c75b14349a481184cd42dd8b8ede6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:21.863350 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d26ff78aa5f94ae49a197daf2be00aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26ff78aa5f94ae49a197daf2be00aa5 already in progress: creating tablet
I20251029 08:52:21.864326 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf9e36c47be4509a998be8e7138475b with delete_type TABLET_DATA_DELETED (replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.864568 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.864720 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.864768 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.864969 758 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf9e36c47be4509a998be8e7138475b with delete_type TABLET_DATA_DELETED (replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.865763 2249 raft_consensus.cc:493] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.865917 2249 raft_consensus.cc:515] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:21.866709 2249 leader_election.cc:290] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.867573 1183 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.867760 1183 ts_tablet_manager.cc:1403] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
W20251029 08:52:21.867779 998 leader_election.cc:343] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:21.868741 1183 raft_consensus.cc:359] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.868955 1183 raft_consensus.cc:385] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.869024 1183 raft_consensus.cc:740] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:21.869282 996 leader_election.cc:343] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Not found: Tablet not found: 2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:21.869386 996 leader_election.cc:304] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.869742 2249 raft_consensus.cc:2749] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.872102 1179 ts_tablet_manager.cc:1929] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:21.872221 1179 log.cc:1199] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/cdcc9a441c6a4d82a58262f4fdb3f00d
I20251029 08:52:21.872203 1183 consensus_queue.cc:260] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:21.872273 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet dbf9e36c47be4509a998be8e7138475b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf9e36c47be4509a998be8e7138475b already in progress: creating tablet
I20251029 08:52:21.872728 1183 ts_tablet_manager.cc:1434] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:21.873378 1179 ts_tablet_manager.cc:1950] T cdcc9a441c6a4d82a58262f4fdb3f00d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.873785 1183 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.875590 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]) successfully deleted
I20251029 08:52:21.876312 1179 tablet_replica.cc:333] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.876516 1179 raft_consensus.cc:2243] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.876459 2249 consensus_queue.cc:1048] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:21.876657 1179 raft_consensus.cc:2272] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.878298 1179 ts_tablet_manager.cc:1916] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.878652 1178 tablet_bootstrap.cc:654] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.881253 1183 tablet_bootstrap.cc:654] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.891697 1183 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.891885 1183 ts_tablet_manager.cc:1403] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20251029 08:52:21.892287 1176 ts_tablet_manager.cc:1950] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.892772 1183 raft_consensus.cc:359] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.892932 1183 raft_consensus.cc:385] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.892982 1183 raft_consensus.cc:740] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.893164 1183 consensus_queue.cc:260] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.893462 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (table default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]) successfully deleted
I20251029 08:52:21.893687 594 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b41e5505835e43e6b7e714c02b18ec6d
I20251029 08:52:21.894017 1176 tablet_replica.cc:333] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:21.894075 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4189d781b1c24ebaae9225c1c511ec73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.894203 1176 raft_consensus.cc:2243] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.894342 1176 raft_consensus.cc:2272] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.894651 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4189d781b1c24ebaae9225c1c511ec73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.894860 1176 ts_tablet_manager.cc:1916] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:21.895535 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f2f8ee81fa654340ba4535f2a11a0fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.896750 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c44114212a547b4b4082d590f14e263" } new_table_name: "default.Table_47d202b2ab454856a15f3b5705fc9188" modify_external_catalogs: true
I20251029 08:52:21.897557 1178 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.897692 1178 ts_tablet_manager.cc:1403] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20251029 08:52:21.897961 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:21.898236 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.899178 1178 raft_consensus.cc:359] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.899328 1178 raft_consensus.cc:385] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.899369 1178 raft_consensus.cc:740] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.900249 1178 consensus_queue.cc:260] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.903270 1179 ts_tablet_manager.cc:1929] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.903851 1178 ts_tablet_manager.cc:1434] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:21.904814 1178 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.906450 1178 tablet_bootstrap.cc:654] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.909587 1183 ts_tablet_manager.cc:1434] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20251029 08:52:21.912565 1179 log.cc:1199] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bda3e143699b41b6bba308c83bbb3e0d
I20251029 08:52:21.912899 1183 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.913611 2290 raft_consensus.cc:493] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.913802 1179 ts_tablet_manager.cc:1950] T bda3e143699b41b6bba308c83bbb3e0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.913748 2290 raft_consensus.cc:515] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.914348 2290 leader_election.cc:290] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:21.914789 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:21.914901 2290 raft_consensus.cc:493] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:21.915016 2290 raft_consensus.cc:515] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.915004 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6c752dc40c034ebaa141deb809297b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.915498 2290 leader_election.cc:290] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:21.915593 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c752dc40c034ebaa141deb809297b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.915956 1183 tablet_bootstrap.cc:654] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.915958 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bda3e143699b41b6bba308c83bbb3e0d (table default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]) successfully deleted
I20251029 08:52:21.916420 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.916553 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "254e62d88c3242a0a3ac7470a8005d74" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:21.917155 720 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.917230 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:21.917397 720 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.917577 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "254e62d88c3242a0a3ac7470a8005d74" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:21.917986 1179 tablet_replica.cc:333] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:21.918129 718 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.918227 718 leader_election.cc:304] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.918239 1179 raft_consensus.cc:2243] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.918401 1179 raft_consensus.cc:2272] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.918592 718 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:21.918690 718 leader_election.cc:304] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:21.919000 2289 raft_consensus.cc:2749] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.919142 2289 raft_consensus.cc:2749] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:21.919988 1179 ts_tablet_manager.cc:1916] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:21.922166 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00829c7fa0fe4e379a1a3044550e5941 consensus state has no leader
I20251029 08:52:21.922631 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1ca0aca0e954427d84908b738587abdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.926690 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_588b8eaf2cad4b739475473bec43d9cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.928493 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_096667619d1847a998aa237bb19f9024"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:21.930502 1179 ts_tablet_manager.cc:1929] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.930621 1179 log.cc:1199] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2f40aafe7dc94851a67abd84d80d7d7b
I20251029 08:52:21.931020 1179 ts_tablet_manager.cc:1950] T 2f40aafe7dc94851a67abd84d80d7d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.932395 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2f40aafe7dc94851a67abd84d80d7d7b (table default.Table_b702ba0aca804d06837f0aa80bae8669 [id=22cd3b05e2c74671a965cd770dcee5a9]) successfully deleted
W20251029 08:52:21.932672 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_096667619d1847a998aa237bb19f9024 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.932792 1179 tablet_replica.cc:333] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.932994 1179 raft_consensus.cc:2243] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.933146 1179 raft_consensus.cc:2272] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:21.933130 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ca0aca0e954427d84908b738587abdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.934008 1179 ts_tablet_manager.cc:1916] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:21.934262 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_588b8eaf2cad4b739475473bec43d9cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:21.942286 1179 ts_tablet_manager.cc:1929] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.942358 1176 ts_tablet_manager.cc:1929] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:21.942505 1179 log.cc:1199] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/1ae561a995c94fadb288b6395c55a1bf
I20251029 08:52:21.943259 1179 ts_tablet_manager.cc:1950] T 1ae561a995c94fadb288b6395c55a1bf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:21.944435 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 1ae561a995c94fadb288b6395c55a1bf (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:21.945011 2295 leader_election.cc:290] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:21.945068 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3ba512bd4f04bd9aceb4cb210fe9689" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:21.945254 1071 raft_consensus.cc:2468] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:21.945463 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a4ae5978b71443a48a88bc69128b187d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4ae5978b71443a48a88bc69128b187d already in progress: creating tablet
I20251029 08:52:21.945695 861 leader_election.cc:304] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:21.946257 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet dbf9e36c47be4509a998be8e7138475b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf9e36c47be4509a998be8e7138475b already in progress: creating tablet
I20251029 08:52:21.946820 1176 log.cc:1199] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:21.947181 1176 ts_tablet_manager.cc:1950] T d50e2fdfc0e64516a41c90f5a56f4c89 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:21.948987 2295 raft_consensus.cc:2804] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:21.949105 2295 raft_consensus.cc:493] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:21.949251 2295 raft_consensus.cc:3060] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.943454 1178 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:21.955552 1178 ts_tablet_manager.cc:1403] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20251029 08:52:21.956524 1178 raft_consensus.cc:359] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.956765 1178 raft_consensus.cc:385] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.956877 1178 raft_consensus.cc:740] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.957175 1178 consensus_queue.cc:260] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.958031 1178 ts_tablet_manager.cc:1434] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:21.964488 1179 tablet_replica.cc:333] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:21.964730 1179 raft_consensus.cc:2243] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:21.964897 1179 raft_consensus.cc:2272] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:21.965660 1178 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:21.967960 1178 tablet_bootstrap.cc:654] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.978900 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0d08838d55e248ba99e00532bb82b65f
I20251029 08:52:21.980618 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d08838d55e248ba99e00532bb82b65f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:21.980998 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d08838d55e248ba99e00532bb82b65f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:21.981719 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0d08838d55e248ba99e00532bb82b65f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d08838d55e248ba99e00532bb82b65f already in progress: creating tablet
I20251029 08:52:21.984660 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 82c522c0b20647b296d0439773a95aa6
I20251029 08:52:21.985579 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 82c522c0b20647b296d0439773a95aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.986425 1183 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:21.986609 1183 ts_tablet_manager.cc:1403] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.002s
I20251029 08:52:21.986793 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 13afd85e234643e08957e7a7f7fd7e22
I20251029 08:52:21.987269 588 catalog_manager.cc:6719] ReplaceTablet: tablet 13afd85e234643e08957e7a7f7fd7e22 deleted and replaced by tablet f9ea32f5eecf41a39925cdd5c07f9a88
I20251029 08:52:21.987677 1183 raft_consensus.cc:359] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.987946 1183 raft_consensus.cc:385] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:21.988010 1183 raft_consensus.cc:740] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:21.988963 1183 consensus_queue.cc:260] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:21.989461 1183 ts_tablet_manager.cc:1434] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251029 08:52:21.989804 1183 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:21.991941 1183 tablet_bootstrap.cc:654] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:21.992117 2295 raft_consensus.cc:515] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:21.993880 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 82c522c0b20647b296d0439773a95aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c522c0b20647b296d0439773a95aa6 already in progress: creating tablet
I20251029 08:52:21.993971 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 with delete_type TABLET_DATA_DELETED (replaced by tablet f9ea32f5eecf41a39925cdd5c07f9a88) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:21.994113 2295 leader_election.cc:290] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:21.994472 861 leader_election.cc:343] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a3ba512bd4f04bd9aceb4cb210fe9689
W20251029 08:52:21.994737 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 13afd85e234643e08957e7a7f7fd7e22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13afd85e234643e08957e7a7f7fd7e22 already in progress: creating tablet
I20251029 08:52:21.994904 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3ba512bd4f04bd9aceb4cb210fe9689" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:21.995131 1071 raft_consensus.cc:3060] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:21.994904 1179 ts_tablet_manager.cc:1916] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.000217 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_d64ea78b6d1648eabf348d4da0cffae8 [id=32fdba980db34781a7f359080cb1e56c]). Expected version 1 got 0
I20251029 08:52:22.001098 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 82c522c0b20647b296d0439773a95aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.001111 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 with delete_type TABLET_DATA_DELETED (replaced by tablet f9ea32f5eecf41a39925cdd5c07f9a88) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.002135 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.003523 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.004985 2290 raft_consensus.cc:493] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.005147 2290 raft_consensus.cc:515] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.005769 2290 leader_election.cc:290] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.006554 889 tablet_service.cc:1552] Processing DeleteTablet for tablet afa2bb1075c4418eb10f4493f5e73c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.006781 1974 tablet_replica.cc:333] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:22.006942 1974 raft_consensus.cc:2243] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.007110 1974 raft_consensus.cc:2272] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.007282 891 tablet_service.cc:1552] Processing DeleteTablet for tablet bda3e143699b41b6bba308c83bbb3e0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25bd12c9aa01495e83b09e1d82da20ad) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.007838 1974 ts_tablet_manager.cc:1916] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.010619 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b58dc38f334840779d240a10ccb0d8db" } new_table_name: "default.Table_1d7a8b08b5104bb2afd66ad8dec95210" modify_external_catalogs: true
I20251029 08:52:22.012785 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.013103 2294 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
W20251029 08:52:22.013906 718 leader_election.cc:343] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.016662 2269 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20251029 08:52:22.020038 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.020239 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.021550 753 tablet_service.cc:1505] Processing CreateTablet for tablet 37dfd4633604497d8c4d1760f33e5db7 (DEFAULT_TABLE table=default.Table_991230beb13844e8b60b5cdcc634a480 [id=77c5185d7e1e45379932c993b5d191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.023507 745 tablet_service.cc:1505] Processing CreateTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea (DEFAULT_TABLE table=default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9 [id=c7d8cbafe1624361a7551cb904b7cf6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.027312 748 tablet_service.cc:1505] Processing CreateTablet for tablet f062556d67a148d19d242b8d99cfdff9 (DEFAULT_TABLE table=default.Table_0d4cf693087a4d308c9e97704e60754d [id=ef7dd7e7e527423e8938cd03b25310ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.027869 757 tablet_service.cc:1505] Processing CreateTablet for tablet 15686669f6f44ee7b2609899d13556fd (DEFAULT_TABLE table=default.Table_1a22395ba73c42eab774fb043c38da7b [id=8546c3a373444ebb89a53b4f92306904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.028311 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.029982 749 tablet_service.cc:1505] Processing CreateTablet for tablet f9d6d7b3e06144cbb1cabec1a9360340 (DEFAULT_TABLE table=default.Table_2a5ef8abeebf415084663ef00f8d942b [id=774a0601f9414698aa5a672a5df1eed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.030346 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.033533 758 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd82e3549442ada2503518037127fc (DEFAULT_TABLE table=default.Table_8c17839d47d445acb4e7f4d30d5850dc [id=01265c9a17ea4607ab22c232d413741a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.033629 896 tablet_service.cc:1505] Processing CreateTablet for tablet 15686669f6f44ee7b2609899d13556fd (DEFAULT_TABLE table=default.Table_1a22395ba73c42eab774fb043c38da7b [id=8546c3a373444ebb89a53b4f92306904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.033635 2361 consensus_queue.cc:1048] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:22.033823 895 tablet_service.cc:1505] Processing CreateTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea (DEFAULT_TABLE table=default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9 [id=c7d8cbafe1624361a7551cb904b7cf6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.033635 902 tablet_service.cc:1505] Processing CreateTablet for tablet 37dfd4633604497d8c4d1760f33e5db7 (DEFAULT_TABLE table=default.Table_991230beb13844e8b60b5cdcc634a480 [id=77c5185d7e1e45379932c993b5d191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.034281 892 tablet_service.cc:1505] Processing CreateTablet for tablet f062556d67a148d19d242b8d99cfdff9 (DEFAULT_TABLE table=default.Table_0d4cf693087a4d308c9e97704e60754d [id=ef7dd7e7e527423e8938cd03b25310ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.034824 887 tablet_service.cc:1505] Processing CreateTablet for tablet f9d6d7b3e06144cbb1cabec1a9360340 (DEFAULT_TABLE table=default.Table_2a5ef8abeebf415084663ef00f8d942b [id=774a0601f9414698aa5a672a5df1eed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.035346 890 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd82e3549442ada2503518037127fc (DEFAULT_TABLE table=default.Table_8c17839d47d445acb4e7f4d30d5850dc [id=01265c9a17ea4607ab22c232d413741a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.036047 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 351047c725f8488abbd074bd40790b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.036500 891 tablet_service.cc:1552] Processing DeleteTablet for tablet a3ba512bd4f04bd9aceb4cb210fe9689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.038928 2361 consensus_queue.cc:1048] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:22.040373 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet aa348d6fa8c644bd8fe7b297a53c1251
I20251029 08:52:22.040681 891 tablet_service.cc:1505] Processing CreateTablet for tablet f4cb38b4023242fa8e85356dd4cccd70 (DEFAULT_TABLE table=default.Table_45ce89ee1267486fb1dc5d6990cb0716 [id=6cfbfe11dc1440cea62e314f0048743a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.041678 747 tablet_service.cc:1552] Processing DeleteTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.045768 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_9357468257554705827bd3bc1eee7bbc" } new_table_name: "default.Table_4e81a355a1424094aa3b482cce58897a" modify_external_catalogs: true
I20251029 08:52:22.047139 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c2da7feabcf14a2f9d60e558d438633f" } new_table_name: "default.Table_769cc9a7a9124054bf155522691740b5" modify_external_catalogs: true
I20251029 08:52:22.049494 889 tablet_service.cc:1552] Processing DeleteTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.052621 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b7a6008b4e6243ccaf166a31f1a9f62c
W20251029 08:52:22.054049 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 24ab9df6f768410eb4f66cffdcd68df2 (table default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24ab9df6f768410eb4f66cffdcd68df2 consensus state has no leader
I20251029 08:52:22.055503 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.054049 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:22.058370 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15686669f6f44ee7b2609899d13556fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.058801 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37dfd4633604497d8c4d1760f33e5db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.058878 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37dfd4633604497d8c4d1760f33e5db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.059250 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d09ed524e97c4c1bb3c4aaea876415ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.059321 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d09ed524e97c4c1bb3c4aaea876415ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.059641 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f062556d67a148d19d242b8d99cfdff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.059871 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd82e3549442ada2503518037127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.060081 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d6d7b3e06144cbb1cabec1a9360340. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.060441 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd82e3549442ada2503518037127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.061054 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cb38b4023242fa8e85356dd4cccd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.061029 1071 raft_consensus.cc:2468] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:22.063221 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:22.063736 1177 tablet_bootstrap.cc:492] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.063881 1177 ts_tablet_manager.cc:1403] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.259s user 0.021s sys 0.034s
I20251029 08:52:22.064478 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15686669f6f44ee7b2609899d13556fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.064988 861 leader_election.cc:304] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:22.065258 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f062556d67a148d19d242b8d99cfdff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.066131 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d6d7b3e06144cbb1cabec1a9360340. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.066517 2199 raft_consensus.cc:2804] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:22.068169 2199 raft_consensus.cc:697] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:22.068439 2199 consensus_queue.cc:237] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.079002 1177 raft_consensus.cc:359] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.079219 1177 raft_consensus.cc:385] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.079272 1177 raft_consensus.cc:740] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.079960 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_480cdda5b5e84032a978c7065e456721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.080678 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_480cdda5b5e84032a978c7065e456721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.081987 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b702ba0aca804d06837f0aa80bae8669" } new_table_name: "default.Table_1c14de238c434c4084a2b70bdf36d9a1" modify_external_catalogs: true
I20251029 08:52:22.084874 2289 raft_consensus.cc:493] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.085024 2289 raft_consensus.cc:515] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.085693 2289 leader_election.cc:290] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.085975 1177 consensus_queue.cc:260] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.088842 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5ac1af8b78a4d94b0e3da14ebfffd92" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.089143 889 tablet_service.cc:1552] Processing DeleteTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.089720 1177 ts_tablet_manager.cc:1434] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
W20251029 08:52:22.090130 718 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:22.091825 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb22af7515084e099b41f9c1c92c067a consensus state has no leader
I20251029 08:52:22.094116 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630] alter complete (version 7)
I20251029 08:52:22.099684 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_8cfe61c96c824cf4adcc750f24156238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.100428 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cfe61c96c824cf4adcc750f24156238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:22.101616 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55ec37d15bb64edf8d6552ee068d3573 consensus state has no leader
I20251029 08:52:22.101887 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d15768ec762b4df6ad8b993fad1087a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.102447 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d15768ec762b4df6ad8b993fad1087a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.107446 1176 tablet_replica.cc:333] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:22.107661 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1a0289f2e27f44a3a7089b80c844d2d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:22.108379 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d50e2fdfc0e64516a41c90f5a56f4c89 (table default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]) successfully deleted
W20251029 08:52:22.109212 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a0289f2e27f44a3a7089b80c844d2d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.108326 1176 raft_consensus.cc:2243] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.109838 1176 raft_consensus.cc:2272] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.111665 1176 ts_tablet_manager.cc:1916] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:22.111802 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15dae38bcced496e90ca8eec280a8317 consensus state has no leader
I20251029 08:52:22.114854 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b785a13a53ea43f6a259db613714c2c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.115439 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b785a13a53ea43f6a259db613714c2c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.119093 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f713c7b9b580405c87b7fdc7df46acf4" } modify_external_catalogs: true
I20251029 08:52:22.119988 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f713c7b9b580405c87b7fdc7df46acf4" } modify_external_catalogs: true
I20251029 08:52:22.124661 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d56c826be1cc4fc1a9a8c42bf71799b1" } new_table_name: "default.Table_6dd58dc30f8f46eeb485ead12c484319" modify_external_catalogs: true
I20251029 08:52:22.127602 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_de611c4a46854ca195eb3cd4001470bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.128499 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de611c4a46854ca195eb3cd4001470bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.130110 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4a2a813a43cf444ab957477fa46cb928"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.130645 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a2a813a43cf444ab957477fa46cb928 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.131587 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 63e705a0b5264eb7a6988129edbf5970
I20251029 08:52:22.132858 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 63e705a0b5264eb7a6988129edbf5970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.134819 1177 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.135031 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 63e705a0b5264eb7a6988129edbf5970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:22.142867 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 169778us
I20251029 08:52:22.146899 1183 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.147056 1183 ts_tablet_manager.cc:1403] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.157s user 0.004s sys 0.003s
I20251029 08:52:22.147827 1183 raft_consensus.cc:359] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.148000 1183 raft_consensus.cc:385] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.148095 1183 raft_consensus.cc:740] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.148288 1183 consensus_queue.cc:260] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.148733 1183 ts_tablet_manager.cc:1434] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:22.149432 1183 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.150492 1974 ts_tablet_manager.cc:1929] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:22.150593 1974 log.cc:1199] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/afa2bb1075c4418eb10f4493f5e73c1b
I20251029 08:52:22.150890 1974 ts_tablet_manager.cc:1950] T afa2bb1075c4418eb10f4493f5e73c1b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:22.151218 2269 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:22.151388 1179 ts_tablet_manager.cc:1929] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:22.151475 1179 log.cc:1199] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:22.151835 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d08838d55e248ba99e00532bb82b65f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.152262 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 82c522c0b20647b296d0439773a95aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.152572 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 13afd85e234643e08957e7a7f7fd7e22 with delete_type TABLET_DATA_DELETED (replaced by tablet f9ea32f5eecf41a39925cdd5c07f9a88) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.152819 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.153273 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 37dfd4633604497d8c4d1760f33e5db7 (DEFAULT_TABLE table=default.Table_991230beb13844e8b60b5cdcc634a480 [id=77c5185d7e1e45379932c993b5d191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.153957 1036 tablet_service.cc:1505] Processing CreateTablet for tablet f062556d67a148d19d242b8d99cfdff9 (DEFAULT_TABLE table=default.Table_0d4cf693087a4d308c9e97704e60754d [id=ef7dd7e7e527423e8938cd03b25310ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.154425 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 15686669f6f44ee7b2609899d13556fd (DEFAULT_TABLE table=default.Table_1a22395ba73c42eab774fb043c38da7b [id=8546c3a373444ebb89a53b4f92306904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.154495 1035 tablet_service.cc:1505] Processing CreateTablet for tablet f9d6d7b3e06144cbb1cabec1a9360340 (DEFAULT_TABLE table=default.Table_2a5ef8abeebf415084663ef00f8d942b [id=774a0601f9414698aa5a672a5df1eed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.154812 1050 tablet_service.cc:1505] Processing CreateTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea (DEFAULT_TABLE table=default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9 [id=c7d8cbafe1624361a7551cb904b7cf6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.155046 1039 tablet_service.cc:1505] Processing CreateTablet for tablet f4cb38b4023242fa8e85356dd4cccd70 (DEFAULT_TABLE table=default.Table_45ce89ee1267486fb1dc5d6990cb0716 [id=6cfbfe11dc1440cea62e314f0048743a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.155297 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 25bd82e3549442ada2503518037127fc (DEFAULT_TABLE table=default.Table_8c17839d47d445acb4e7f4d30d5850dc [id=01265c9a17ea4607ab22c232d413741a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.155393 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.155771 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.155983 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 63e705a0b5264eb7a6988129edbf5970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.156363 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.155313 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet aa348d6fa8c644bd8fe7b297a53c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.156775 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5ac1af8b78a4d94b0e3da14ebfffd92" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.157449 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37dfd4633604497d8c4d1760f33e5db7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:22.157552 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a866b5ea4d6e4e89a5a5bf4f82bc6e2a consensus state has no leader
I20251029 08:52:22.159552 2289 raft_consensus.cc:493] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.159686 2289 raft_consensus.cc:515] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.160487 2289 leader_election.cc:290] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.162154 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9ebc2ba31e644183b47973a987d1d9b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.162736 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ebc2ba31e644183b47973a987d1d9b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.165724 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:22.167066 1974 tablet_replica.cc:333] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:22.167249 1974 raft_consensus.cc:2243] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:22.167480 1974 pending_rounds.cc:70] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:22.167531 1974 pending_rounds.cc:77] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037649201549312 op_type: NO_OP noop_request { }
I20251029 08:52:22.167613 1974 raft_consensus.cc:2889] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:22.167704 1974 raft_consensus.cc:2272] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.168393 2289 raft_consensus.cc:493] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.168490 1974 ts_tablet_manager.cc:1916] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.168529 2289 raft_consensus.cc:515] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.169128 2289 leader_election.cc:290] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.169411 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:22.169994 2294 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:22.173298 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f062556d67a148d19d242b8d99cfdff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.176266 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15686669f6f44ee7b2609899d13556fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.151880 1179 ts_tablet_manager.cc:1950] T b7da7e08b74f4a9aacedada42e839a11 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:22.176985 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_def435193a6b4fb893c46c0029059885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.177600 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_def435193a6b4fb893c46c0029059885 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.178589 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e213713757949caa3f504f70026b7b5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.178926 891 tablet_service.cc:1552] Processing DeleteTablet for tablet b7da7e08b74f4a9aacedada42e839a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.180493 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bbf6401423411089c9070f3b03f1a7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.180920 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet afa2bb1075c4418eb10f4493f5e73c1b (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:22.181399 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f760df7015424091acf5d9b6b39bfbd3" } modify_external_catalogs: true
I20251029 08:52:22.181593 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f760df7015424091acf5d9b6b39bfbd3" } modify_external_catalogs: true
I20251029 08:52:22.181936 2366 consensus_queue.cc:1048] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:22.182756 720 leader_election.cc:343] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.180325 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d6d7b3e06144cbb1cabec1a9360340. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.182900 720 leader_election.cc:304] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.183593 2289 raft_consensus.cc:2749] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:22.185496 718 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:22.185858 718 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.188124 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d09ed524e97c4c1bb3c4aaea876415ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.188572 1183 tablet_bootstrap.cc:654] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:22.188923 720 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.189025 720 leader_election.cc:304] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.189383 2289 raft_consensus.cc:2749] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.201489 1177 tablet_bootstrap.cc:654] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.205109 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_626c2d6e7b6b4dea95389dd85a113864" } modify_external_catalogs: true
I20251029 08:52:22.205421 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_626c2d6e7b6b4dea95389dd85a113864" } modify_external_catalogs: true
I20251029 08:52:22.211475 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_78669d09485b482a90061cad43238368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.212451 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78669d09485b482a90061cad43238368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.224138 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cb38b4023242fa8e85356dd4cccd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.224982 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.225343 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e213713757949caa3f504f70026b7b5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:22.226310 720 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.226467 720 leader_election.cc:304] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.226955 2289 raft_consensus.cc:2749] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.230127 1177 tablet_bootstrap.cc:492] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.230270 1177 ts_tablet_manager.cc:1403] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.001s
I20251029 08:52:22.230939 1177 raft_consensus.cc:359] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.231104 1177 raft_consensus.cc:385] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.231174 1177 raft_consensus.cc:740] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.231937 1177 consensus_queue.cc:260] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.232956 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bbf6401423411089c9070f3b03f1a7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:22.233677 720 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.233798 720 leader_election.cc:304] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.234153 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bd82e3549442ada2503518037127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.234180 2289 raft_consensus.cc:2749] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.235435 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b582dfeab662456ba4730f5c80455d83
I20251029 08:52:22.236672 748 tablet_service.cc:1552] Processing DeleteTablet for tablet b582dfeab662456ba4730f5c80455d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.238343 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet b582dfeab662456ba4730f5c80455d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.238619 1177 ts_tablet_manager.cc:1434] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:22.240128 891 tablet_service.cc:1552] Processing DeleteTablet for tablet b582dfeab662456ba4730f5c80455d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.241657 1177 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.242895 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c407207334e644189fdbfcae1dc12ba9" } modify_external_catalogs: true
I20251029 08:52:22.243125 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c407207334e644189fdbfcae1dc12ba9" } modify_external_catalogs: true
I20251029 08:52:22.244490 1179 tablet_replica.cc:333] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:22.244709 1179 raft_consensus.cc:2243] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.245234 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b7da7e08b74f4a9aacedada42e839a11 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:22.247162 1177 tablet_bootstrap.cc:654] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.261019 1179 raft_consensus.cc:2272] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.262394 1178 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:22.262573 1178 ts_tablet_manager.cc:1403] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.298s user 0.004s sys 0.006s
I20251029 08:52:22.263417 1178 raft_consensus.cc:359] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.263679 1178 raft_consensus.cc:385] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.263731 1178 raft_consensus.cc:740] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.265177 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9" } modify_external_catalogs: true
I20251029 08:52:22.265341 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57d82fc61e864d3c8d43be4bb9ee2ad9" } modify_external_catalogs: true
I20251029 08:52:22.265899 1179 ts_tablet_manager.cc:1916] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.269214 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ebcdf239d5db4bc4bb61f2faf30e05d1" } new_table_name: "default.Table_d321f6bf95ce4764a4617e142a96f7cb" modify_external_catalogs: true
I20251029 08:52:22.270539 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0b61c710897f43f798d88d92c4676d0d
I20251029 08:52:22.271687 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b61c710897f43f798d88d92c4676d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.272495 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b61c710897f43f798d88d92c4676d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.273654 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b61c710897f43f798d88d92c4676d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.273763 748 tablet_service.cc:1505] Processing CreateTablet for tablet f4cb38b4023242fa8e85356dd4cccd70 (DEFAULT_TABLE table=default.Table_45ce89ee1267486fb1dc5d6990cb0716 [id=6cfbfe11dc1440cea62e314f0048743a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.277082 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 36bea657aa9044f88e50bf4e749ff01e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:22.278220 2394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40435 every 8 connection(s)
I20251029 08:52:22.280310 2066 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40435
I20251029 08:52:22.281152 2380 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:22.283363 2066 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/master-0/data/info.pb
I20251029 08:52:22.283818 2367 raft_consensus.cc:493] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.283955 2367 raft_consensus.cc:515] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.284816 891 tablet_service.cc:1505] Processing CreateTablet for tablet 72a9f105b54d4e828487ecbcbeb35764 (DEFAULT_TABLE table=default.Table_020f3afc90994322af7afdfce26b6e7f [id=d80e9b086786428b951c872cf343f310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.285223 2367 leader_election.cc:290] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:22.286309 998 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:22.286486 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu as pid 2066
I20251029 08:52:22.286490 749 tablet_service.cc:1505] Processing CreateTablet for tablet 72a9f105b54d4e828487ecbcbeb35764 (DEFAULT_TABLE table=default.Table_020f3afc90994322af7afdfce26b6e7f [id=d80e9b086786428b951c872cf343f310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.287295 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c66cbb495f5142d79a674f7d5adc505e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.287796 896 tablet_service.cc:1505] Processing CreateTablet for tablet adb59d3ee338465a942e6b4423f0e071 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.287966 1044 tablet_service.cc:1505] Processing CreateTablet for tablet adb59d3ee338465a942e6b4423f0e071 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.291306 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cb38b4023242fa8e85356dd4cccd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.291956 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9f105b54d4e828487ecbcbeb35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.292380 745 tablet_service.cc:1505] Processing CreateTablet for tablet adb59d3ee338465a942e6b4423f0e071 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.292697 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb59d3ee338465a942e6b4423f0e071. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:22.293644 996 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.294656 2367 raft_consensus.cc:493] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.294890 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 72a9f105b54d4e828487ecbcbeb35764 (DEFAULT_TABLE table=default.Table_020f3afc90994322af7afdfce26b6e7f [id=d80e9b086786428b951c872cf343f310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.295717 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb59d3ee338465a942e6b4423f0e071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.295842 996 leader_election.cc:304] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.296995 2359 raft_consensus.cc:2749] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.300137 1974 ts_tablet_manager.cc:1929] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:22.300285 1974 log.cc:1199] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bda3e143699b41b6bba308c83bbb3e0d
I20251029 08:52:22.300684 1974 ts_tablet_manager.cc:1950] T bda3e143699b41b6bba308c83bbb3e0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:22.301028 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9f105b54d4e828487ecbcbeb35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.304538 1177 tablet_bootstrap.cc:492] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.304675 1177 ts_tablet_manager.cc:1403] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.009s
I20251029 08:52:22.305471 1177 raft_consensus.cc:359] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.305636 1177 raft_consensus.cc:385] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.305688 1177 raft_consensus.cc:740] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.306051 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb59d3ee338465a942e6b4423f0e071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.311229 895 tablet_service.cc:1505] Processing CreateTablet for tablet 19ab7e172a704db3aa4c32da35993003 (DEFAULT_TABLE table=default.Table_24dc6417d2284eb380c010cc2586067c [id=335eeadc8a574dce8391f30f245d89d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.311570 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ab7e172a704db3aa4c32da35993003. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.317961 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7ee09f96373f4bb19ffe77904d76cb95" } new_table_name: "default.Table_d25e6fa400a94d7ca4b9b5c566688235" modify_external_catalogs: true
I20251029 08:52:22.319314 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_acdad95817bc4aaa86670404d517c37d" } new_table_name: "default.Table_b58caa6aed5542948934aeca3e67d32d" modify_external_catalogs: true
I20251029 08:52:22.263964 1178 consensus_queue.cc:260] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.320587 1178 ts_tablet_manager.cc:1434] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.058s user 0.003s sys 0.000s
I20251029 08:52:22.320690 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_122ac766040a47f39ba5012bc56c5029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:22.320878 1178 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:22.321183 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_122ac766040a47f39ba5012bc56c5029 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.325001 1177 consensus_queue.cc:260] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.326157 2380 tablet.cc:1722] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20251029 08:52:22.327189 1177 ts_tablet_manager.cc:1434] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20251029 08:52:22.327467 1177 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.332056 1974 tablet_replica.cc:333] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:22.332263 1974 raft_consensus.cc:2243] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:22.332304 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bda3e143699b41b6bba308c83bbb3e0d (table default.Table_06bcd895f6814a55a1dbda4c7b9e1acc [id=f016a97fbabf478aa5228bb481ac468f]) successfully deleted
I20251029 08:52:22.332485 1974 pending_rounds.cc:70] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:22.332533 1974 pending_rounds.cc:77] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037650218496000 op_type: NO_OP noop_request { }
I20251029 08:52:22.332617 1974 raft_consensus.cc:2889] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:22.332713 1974 raft_consensus.cc:2272] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.332969 1177 tablet_bootstrap.cc:654] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.335392 1183 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.335553 1183 ts_tablet_manager.cc:1403] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.186s user 0.005s sys 0.002s
I20251029 08:52:22.336316 1183 raft_consensus.cc:359] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.336530 1183 raft_consensus.cc:385] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.336597 1183 raft_consensus.cc:740] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.338793 1974 ts_tablet_manager.cc:1916] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.339534 1183 consensus_queue.cc:260] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.342152 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9f105b54d4e828487ecbcbeb35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.342515 2199 raft_consensus.cc:493] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.342644 2199 raft_consensus.cc:515] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.343183 2199 leader_election.cc:290] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:22.344147 861 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:22.344722 2367 raft_consensus.cc:515] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.345829 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.346000 920 raft_consensus.cc:2468] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:22.348796 1178 tablet_bootstrap.cc:654] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.360157 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.360257 1177 tablet_bootstrap.cc:492] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.360410 1177 ts_tablet_manager.cc:1403] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.008s
I20251029 08:52:22.361208 1177 raft_consensus.cc:359] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.361383 1177 raft_consensus.cc:385] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.361441 1177 raft_consensus.cc:740] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.361639 1177 consensus_queue.cc:260] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.362111 1177 ts_tablet_manager.cc:1434] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:22.362377 1177 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.364146 1177 tablet_bootstrap.cc:654] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.371865 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9499d0099d1c4edc843fb8d29ca3b1b6
I20251029 08:52:22.372182 1178 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:22.372323 1178 ts_tablet_manager.cc:1403] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20251029 08:52:22.373086 1178 raft_consensus.cc:359] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.373214 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d09ed524e97c4c1bb3c4aaea876415ea
I20251029 08:52:22.373248 1178 raft_consensus.cc:385] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.373299 1178 raft_consensus.cc:740] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.375456 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.376502 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.376727 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.379024 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_1d7a8b08b5104bb2afd66ad8dec95210 [id=01723803a27d472b90b05ba626327d7c] alter complete (version 1)
I20251029 08:52:22.382508 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.383080 1178 consensus_queue.cc:260] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.390753 1178 ts_tablet_manager.cc:1434] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251029 08:52:22.394613 748 tablet_service.cc:1505] Processing CreateTablet for tablet 19ab7e172a704db3aa4c32da35993003 (DEFAULT_TABLE table=default.Table_24dc6417d2284eb380c010cc2586067c [id=335eeadc8a574dce8391f30f245d89d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.395175 1178 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:22.397724 1178 tablet_bootstrap.cc:654] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.397992 2289 raft_consensus.cc:493] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.398109 2289 raft_consensus.cc:515] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.398729 2289 leader_election.cc:290] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.407189 2367 leader_election.cc:290] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.407773 1183 ts_tablet_manager.cc:1434] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 08:52:22.408596 895 tablet_service.cc:1505] Processing CreateTablet for tablet a70b8c594aa24f37a169d1aa29c2268e (DEFAULT_TABLE table=default.Table_3676333781d94b0d80010190fd318da8 [id=4ca4ed2a92744d299fe1b1f7496a1b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.408890 1068 raft_consensus.cc:2468] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:22.409929 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09968d93203140fa9201928c87702ee0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:22.411013 718 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.412048 996 leader_election.cc:304] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.411938 1176 ts_tablet_manager.cc:1929] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:22.412488 1176 log.cc:1199] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/29adc6d53af5419dbda1d41d75d37e71
I20251029 08:52:22.416216 1176 ts_tablet_manager.cc:1950] T 29adc6d53af5419dbda1d41d75d37e71 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:22.416620 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ab7e172a704db3aa4c32da35993003. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.419943 745 tablet_service.cc:1505] Processing CreateTablet for tablet a70b8c594aa24f37a169d1aa29c2268e (DEFAULT_TABLE table=default.Table_3676333781d94b0d80010190fd318da8 [id=4ca4ed2a92744d299fe1b1f7496a1b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.420339 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70b8c594aa24f37a169d1aa29c2268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.421509 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 36bea657aa9044f88e50bf4e749ff01e
I20251029 08:52:22.422150 595 catalog_manager.cc:6719] ReplaceTablet: tablet 36bea657aa9044f88e50bf4e749ff01e deleted and replaced by tablet f004f284a60346dba6252ecd3daf5698
W20251029 08:52:22.423200 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba809e06064141789e7abe6fd3b0567f consensus state has no leader
I20251029 08:52:22.423260 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 36bea657aa9044f88e50bf4e749ff01e with delete_type TABLET_DATA_DELETED (replaced by tablet f004f284a60346dba6252ecd3daf5698) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:22.424310 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63bbad742c4b472fa42745b6550fbbbe consensus state has no leader
I20251029 08:52:22.428953 1177 tablet_bootstrap.cc:492] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.429108 1177 ts_tablet_manager.cc:1403] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.021s
I20251029 08:52:22.429134 891 tablet_service.cc:1505] Processing CreateTablet for tablet 0b61c710897f43f798d88d92c4676d0d (DEFAULT_TABLE table=default.Table_626c2d6e7b6b4dea95389dd85a113864 [id=f0309bf5036d42d48044b4957a4732f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.429544 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 36bea657aa9044f88e50bf4e749ff01e with delete_type TABLET_DATA_DELETED (replaced by tablet f004f284a60346dba6252ecd3daf5698) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.429883 1177 raft_consensus.cc:359] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.430049 1177 raft_consensus.cc:385] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.430107 1177 raft_consensus.cc:740] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.430347 1177 consensus_queue.cc:260] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.430651 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 28d62d1e354d41f49b9ae30dbb6efc0f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:22.432219 2367 raft_consensus.cc:2804] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.432372 2367 raft_consensus.cc:493] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251029 08:52:22.432510 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 64949.8us
I20251029 08:52:22.433279 1177 ts_tablet_manager.cc:1434] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:22.434190 1177 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.435946 1177 tablet_bootstrap.cc:654] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.435909 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3ced4474768d425dae572ad46c65317c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:22.436184 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_038f3279c2364bdeb2dc381a25b699b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.436529 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ced4474768d425dae572ad46c65317c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:22.436538 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038f3279c2364bdeb2dc381a25b699b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.437152 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae561a995c94fadb288b6395c55a1bf with delete_type TABLET_DATA_DELETED (replaced by tablet 9adbcf3f12104fb5bfd1d9dd5c946c54) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.437355 887 tablet_service.cc:1552] Processing DeleteTablet for tablet c66cbb495f5142d79a674f7d5adc505e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.437512 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_007a17f442c74a71a6bb3775642f35fd" } modify_external_catalogs: true
I20251029 08:52:22.437654 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_007a17f442c74a71a6bb3775642f35fd" } modify_external_catalogs: true
I20251029 08:52:22.437744 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1c023b679b4c3298bf953b53d4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet a6d3b48876f04c48b4b2d0ac864ab7e5) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.439000 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ec38d64e3d3455ab090035a692b37ff" } new_table_name: "default.Table_fef23ec2ce374eadbd727deb26c45850" modify_external_catalogs: true
I20251029 08:52:22.439623 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.440474 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ecb4a70de9424d629eaa74f4be4c6549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.440989 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb4a70de9424d629eaa74f4be4c6549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.442216 749 tablet_service.cc:1505] Processing CreateTablet for tablet 0b61c710897f43f798d88d92c4676d0d (DEFAULT_TABLE table=default.Table_626c2d6e7b6b4dea95389dd85a113864 [id=f0309bf5036d42d48044b4957a4732f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.442706 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b61c710897f43f798d88d92c4676d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.447548 2295 raft_consensus.cc:493] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.447710 2295 raft_consensus.cc:515] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.448376 2295 leader_election.cc:290] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:22.449430 861 leader_election.cc:343] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:22.452008 597 master_service.cc:954] ReplaceTablet: received request to replace tablet a2b7853b85d24a04b8b11caaa4fe8c4d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:22.458463 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:22.459564 1177 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.459642 2395 sys_catalog.cc:263] Verifying existing consensus state
I20251029 08:52:22.459739 1177 ts_tablet_manager.cc:1403] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20251029 08:52:22.460395 2289 raft_consensus.cc:493] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.460541 2289 raft_consensus.cc:515] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.461196 2289 leader_election.cc:290] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.462038 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fdc381a0d949b0b93cef399daa3021" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:22.462673 718 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.463263 1177 raft_consensus.cc:359] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.463495 1177 raft_consensus.cc:385] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.463616 1177 raft_consensus.cc:740] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.463979 1177 consensus_queue.cc:260] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:22.464468 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 108518us
I20251029 08:52:22.467272 1177 ts_tablet_manager.cc:1434] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:22.467571 1177 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.468894 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap starting.
I20251029 08:52:22.472801 1177 tablet_bootstrap.cc:654] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.473302 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_65ecb08cc196408da5b8ac8514415da9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.474040 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ecb08cc196408da5b8ac8514415da9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.482838 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 896) took 1005 ms. Trace:
I20251029 08:52:22.483031 1176 rpcz_store.cc:276] 1029 08:52:21.477184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.477249 (+ 65us) service_pool.cc:225] Handling call
1029 08:52:22.107379 (+630130us) ts_tablet_manager.cc:1151] Deleting tablet 29adc6d53af5419dbda1d41d75d37e71
1029 08:52:22.107389 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:22.310561 (+203172us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:22.411379 (+100818us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:22.482810 (+ 71431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":629932}
I20251029 08:52:22.484004 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 29adc6d53af5419dbda1d41d75d37e71 (table default.Table_66d11c1d0bd449aeb31f85ed681c6051 [id=b5a04719c6664c87a647be7fed35ca0f]) successfully deleted
I20251029 08:52:22.483745 1177 tablet_bootstrap.cc:492] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.484082 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 897) took 1006 ms. Trace:
I20251029 08:52:22.484365 1177 ts_tablet_manager.cc:1403] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251029 08:52:22.484464 1176 rpcz_store.cc:276] 1029 08:52:21.477848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.477901 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:22.483916 (+1006015us) ts_tablet_manager.cc:1151] Deleting tablet d26ff78aa5f94ae49a197daf2be00aa5
1029 08:52:22.483927 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:22.484062 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005850}
I20251029 08:52:22.485157 1177 raft_consensus.cc:359] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:22.485250 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d26ff78aa5f94ae49a197daf2be00aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26ff78aa5f94ae49a197daf2be00aa5 already in progress: creating tablet
I20251029 08:52:22.485335 1177 raft_consensus.cc:385] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.485390 1177 raft_consensus.cc:740] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.485596 1177 consensus_queue.cc:260] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.486397 1177 ts_tablet_manager.cc:1434] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:22.486680 1177 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.487322 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 42de32d87ebb451090e0310e36e64277
I20251029 08:52:22.488334 1177 tablet_bootstrap.cc:654] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.488162 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6722a3a5c0714d03b531866b14bfc7db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.488962 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6722a3a5c0714d03b531866b14bfc7db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.490989 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bb3e353385b14cfaa0d01935be38e1e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.491694 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb3e353385b14cfaa0d01935be38e1e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.493399 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3fc2abebc9c249d6b0e46ab667bd9d21" } new_table_name: "default.Table_47a8da9054c74094bd0a2f92bf0dba27" modify_external_catalogs: true
I20251029 08:52:22.496806 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 42de32d87ebb451090e0310e36e64277 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.497109 1177 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.497269 1177 ts_tablet_manager.cc:1403] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251029 08:52:22.497171 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4b2a8ce7300d4dfda4127e313d914f22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.497749 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b2a8ce7300d4dfda4127e313d914f22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.498170 1177 raft_consensus.cc:359] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.498332 1177 raft_consensus.cc:385] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.498384 1177 raft_consensus.cc:740] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.498618 1177 consensus_queue.cc:260] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.499106 1177 ts_tablet_manager.cc:1434] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:22.499382 1177 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:22.499920 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 739f3ef68dc141af90954bf0da4c5d2e consensus state has no leader
I20251029 08:52:22.500360 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11e47ff2a71e439ebfb0e9ba79a95267" } modify_external_catalogs: true
I20251029 08:52:22.500560 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_11e47ff2a71e439ebfb0e9ba79a95267" } modify_external_catalogs: true
I20251029 08:52:22.503036 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 28d62d1e354d41f49b9ae30dbb6efc0f
I20251029 08:52:22.503698 595 catalog_manager.cc:6719] ReplaceTablet: tablet 28d62d1e354d41f49b9ae30dbb6efc0f deleted and replaced by tablet 06b04fdc8080474fbc92fc5606a41c00
I20251029 08:52:22.504434 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f with delete_type TABLET_DATA_DELETED (replaced by tablet 06b04fdc8080474fbc92fc5606a41c00) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.504508 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_036932b8e5bf4baaa335188ce2443357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.505076 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_036932b8e5bf4baaa335188ce2443357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.505861 2269 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:22.506340 1177 tablet_bootstrap.cc:654] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.506831 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f with delete_type TABLET_DATA_DELETED (replaced by tablet 06b04fdc8080474fbc92fc5606a41c00) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.507164 2294 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:22.507398 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 42de32d87ebb451090e0310e36e64277 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.511444 1176 tablet_replica.cc:333] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:22.511703 1176 raft_consensus.cc:2243] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.511945 1176 raft_consensus.cc:2272] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.514487 1176 ts_tablet_manager.cc:1916] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.515796 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b61c710897f43f798d88d92c4676d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.519075 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70b8c594aa24f37a169d1aa29c2268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.519941 1974 ts_tablet_manager.cc:1929] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:22.520154 1974 log.cc:1199] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/351047c725f8488abbd074bd40790b48
I20251029 08:52:22.520594 1974 ts_tablet_manager.cc:1950] T 351047c725f8488abbd074bd40790b48 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:22.522452 2295 raft_consensus.cc:493] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.522578 2295 raft_consensus.cc:515] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.523182 2295 leader_election.cc:290] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.523556 896 tablet_service.cc:1505] Processing CreateTablet for tablet f7f970b620cd40779691fb953e4dd520 (DEFAULT_TABLE table=default.Table_66c918ae6920491b9e2fc399447c8bb5 [id=5b98a07ab61a470997630976464fefd1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:22.524735 861 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:22.525157 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.526517 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:22.526896 1071 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fdc381a0d949b0b93cef399daa3021" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.527357 1183 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.530184 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_14f731cef6024ab79260c04bef01d3b5" } modify_external_catalogs: true
I20251029 08:52:22.530361 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_14f731cef6024ab79260c04bef01d3b5" } modify_external_catalogs: true
I20251029 08:52:22.530331 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 0b61c710897f43f798d88d92c4676d0d (DEFAULT_TABLE table=default.Table_626c2d6e7b6b4dea95389dd85a113864 [id=f0309bf5036d42d48044b4957a4732f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.531407 892 tablet_service.cc:1505] Processing CreateTablet for tablet 97a196d34a4b48988c07034dcc971049 (DEFAULT_TABLE table=default.Table_a4078e8eb124424399751be94d3a19b0 [id=edf465f590564b8b9960229c3b51f8ef]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:22.531549 720 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.531697 720 leader_election.cc:304] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.532135 2289 raft_consensus.cc:2749] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.532418 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:22.532976 745 tablet_service.cc:1505] Processing CreateTablet for tablet 97a196d34a4b48988c07034dcc971049 (DEFAULT_TABLE table=default.Table_a4078e8eb124424399751be94d3a19b0 [id=edf465f590564b8b9960229c3b51f8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.533740 2289 consensus_queue.cc:1048] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:22.534168 887 tablet_service.cc:1505] Processing CreateTablet for tablet 61b901ffd7ae47f487a355f08c076a22 (DEFAULT_TABLE table=default.Table_74466c51bbac4f8a9c086680a8eda1d0 [id=27b95d643e474eb5840a344bf376c728]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.534806 2367 raft_consensus.cc:3060] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.535248 748 tablet_service.cc:1505] Processing CreateTablet for tablet f7f970b620cd40779691fb953e4dd520 (DEFAULT_TABLE table=default.Table_66c918ae6920491b9e2fc399447c8bb5 [id=5b98a07ab61a470997630976464fefd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.535557 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09968d93203140fa9201928c87702ee0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.535837 1178 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
W20251029 08:52:22.535938 998 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:22.535974 1178 ts_tablet_manager.cc:1403] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.001s
W20251029 08:52:22.536314 998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 188570us
W20251029 08:52:22.536465 720 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.536581 720 leader_election.cc:304] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:22.536974 2289 raft_consensus.cc:2749] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.537393 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 61b901ffd7ae47f487a355f08c076a22 (DEFAULT_TABLE table=default.Table_74466c51bbac4f8a9c086680a8eda1d0 [id=27b95d643e474eb5840a344bf376c728]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.538331 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 19ab7e172a704db3aa4c32da35993003 (DEFAULT_TABLE table=default.Table_24dc6417d2284eb380c010cc2586067c [id=335eeadc8a574dce8391f30f245d89d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.538856 749 tablet_service.cc:1505] Processing CreateTablet for tablet 61b901ffd7ae47f487a355f08c076a22 (DEFAULT_TABLE table=default.Table_74466c51bbac4f8a9c086680a8eda1d0 [id=27b95d643e474eb5840a344bf376c728]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.539745 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.540074 1047 tablet_service.cc:1505] Processing CreateTablet for tablet a70b8c594aa24f37a169d1aa29c2268e (DEFAULT_TABLE table=default.Table_3676333781d94b0d80010190fd318da8 [id=4ca4ed2a92744d299fe1b1f7496a1b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.540416 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d09ed524e97c4c1bb3c4aaea876415ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.540597 2359 raft_consensus.cc:493] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.540751 2359 raft_consensus.cc:515] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.540874 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 36bea657aa9044f88e50bf4e749ff01e with delete_type TABLET_DATA_DELETED (replaced by tablet f004f284a60346dba6252ecd3daf5698) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.541090 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 42de32d87ebb451090e0310e36e64277 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.541427 2359 leader_election.cc:290] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.541599 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 28d62d1e354d41f49b9ae30dbb6efc0f with delete_type TABLET_DATA_DELETED (replaced by tablet 06b04fdc8080474fbc92fc5606a41c00) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.541771 1033 tablet_service.cc:1505] Processing CreateTablet for tablet f7f970b620cd40779691fb953e4dd520 (DEFAULT_TABLE table=default.Table_66c918ae6920491b9e2fc399447c8bb5 [id=5b98a07ab61a470997630976464fefd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.541965 890 tablet_service.cc:1505] Processing CreateTablet for tablet b91221923dae416dbf046b555f390d99 (DEFAULT_TABLE table=default.Table_47a0b540f1834bd8bd754a77e504c6e5 [id=3cf39271e2354e568d033bb9b13b0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.542095 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b61c710897f43f798d88d92c4676d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.542116 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 97a196d34a4b48988c07034dcc971049 (DEFAULT_TABLE table=default.Table_a4078e8eb124424399751be94d3a19b0 [id=edf465f590564b8b9960229c3b51f8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.544229 861 leader_election.cc:304] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:22.544689 2295 raft_consensus.cc:2804] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.544767 2295 raft_consensus.cc:493] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.544833 2295 raft_consensus.cc:3060] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.544764 1178 raft_consensus.cc:359] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.544965 1178 raft_consensus.cc:385] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.545039 1178 raft_consensus.cc:740] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.545285 1178 consensus_queue.cc:260] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.545492 2289 raft_consensus.cc:493] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:22.548524 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 919e5494b84343639a2b9b81b05f1c2c consensus state has no leader
I20251029 08:52:22.549576 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41e435e9dfb34d4f8b0c8f7b702d5a6c" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:22.551520 861 leader_election.cc:343] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:22.551628 861 leader_election.cc:304] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:22.551858 2199 raft_consensus.cc:2749] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:22.553310 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 930314c8f90f462990f93fab8f7790c2
I20251029 08:52:22.554292 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 930314c8f90f462990f93fab8f7790c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.557305 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a196d34a4b48988c07034dcc971049. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:22.557356 996 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.557765 2249 raft_consensus.cc:493] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.563668 1046 tablet_service.cc:1505] Processing CreateTablet for tablet b91221923dae416dbf046b555f390d99 (DEFAULT_TABLE table=default.Table_47a0b540f1834bd8bd754a77e504c6e5 [id=3cf39271e2354e568d033bb9b13b0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.563827 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet dbad8c4014b6423cb20e3e5c22ed69f1
I20251029 08:52:22.564872 902 tablet_service.cc:1552] Processing DeleteTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.568002 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 930314c8f90f462990f93fab8f7790c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.568202 2249 raft_consensus.cc:515] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.569451 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f970b620cd40779691fb953e4dd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.570487 902 tablet_service.cc:1505] Processing CreateTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 (DEFAULT_TABLE table=default.Table_f3b73a492eca4e8fa1f1365cbc3518af [id=6b47cbaac80a40f68255258a91d17440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.572297 889 tablet_service.cc:1505] Processing CreateTablet for tablet 5c02b6343368482e88604ed320147762 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.574008 888 tablet_service.cc:1505] Processing CreateTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f (DEFAULT_TABLE table=default.Table_1938936fb1a5483c9d118a57d88d283b [id=a272ed18c02643d59796909106f20396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.574652 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d08838d55e248ba99e00532bb82b65f from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:22.574635 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.574802 922 raft_consensus.cc:2468] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:22.573432 1065 raft_consensus.cc:2468] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:22.575615 2295 raft_consensus.cc:515] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.576162 2295 leader_election.cc:290] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.583544 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5820689813104d31a2a4a5bcf6d6c53a" } modify_external_catalogs: true
I20251029 08:52:22.583781 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5820689813104d31a2a4a5bcf6d6c53a" } modify_external_catalogs: true
I20251029 08:52:22.585183 2249 leader_election.cc:290] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.585443 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 5c02b6343368482e88604ed320147762 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.587011 1183 tablet_bootstrap.cc:654] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.587502 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 (DEFAULT_TABLE table=default.Table_f3b73a492eca4e8fa1f1365cbc3518af [id=6b47cbaac80a40f68255258a91d17440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.588383 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_036932b8e5bf4baaa335188ce2443357" } modify_external_catalogs: true
I20251029 08:52:22.588574 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_036932b8e5bf4baaa335188ce2443357" } modify_external_catalogs: true
I20251029 08:52:22.589753 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5ec3d23d22464070b4557777a2306288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.590277 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ec3d23d22464070b4557777a2306288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.592579 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91221923dae416dbf046b555f390d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.592867 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b901ffd7ae47f487a355f08c076a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.594652 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.594838 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.594880 996 leader_election.cc:304] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.595136 2366 raft_consensus.cc:2804] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.595292 901 tablet_service.cc:1552] Processing DeleteTablet for tablet fe384ca428974ac5ba78e6506a1a8832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.596365 1974 tablet_replica.cc:333] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:22.596446 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b901ffd7ae47f487a355f08c076a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.596573 1974 raft_consensus.cc:2243] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:22.596803 1974 pending_rounds.cc:70] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:22.596846 1974 pending_rounds.cc:77] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037650716483584 op_type: NO_OP noop_request { }
I20251029 08:52:22.596906 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0f1cdfb1804d1c8d6220907795fc80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.596925 1974 raft_consensus.cc:2889] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:22.597011 1974 raft_consensus.cc:2272] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.597311 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f970b620cd40779691fb953e4dd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.597769 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a196d34a4b48988c07034dcc971049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.597625 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:22.598131 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91221923dae416dbf046b555f390d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.598577 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c02b6343368482e88604ed320147762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.598610 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c02b6343368482e88604ed320147762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.598965 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad98112c0ea4f59980e72165cef8a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.600142 2269 tablet.cc:1722] T 1da3e7940352460c9b2bb060174829c9 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:22.606256 1974 ts_tablet_manager.cc:1916] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.606699 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 351047c725f8488abbd074bd40790b48 (table default.Table_04d72f6fb117444a96f5d646217085e2 [id=de08f5f1950e4352a63c7d056bf4ce86]) successfully deleted
I20251029 08:52:22.607256 861 leader_election.cc:304] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:22.607836 891 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.608084 2295 raft_consensus.cc:2804] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.608215 2295 raft_consensus.cc:493] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.608314 2295 raft_consensus.cc:3060] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.610702 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 1da3e7940352460c9b2bb060174829c9 (table default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68]). Expected version 1 got 0
I20251029 08:52:22.614889 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0f1cdfb1804d1c8d6220907795fc80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.615105 2366 raft_consensus.cc:493] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.615203 2366 raft_consensus.cc:3060] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.604344 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f (DEFAULT_TABLE table=default.Table_1938936fb1a5483c9d118a57d88d283b [id=a272ed18c02643d59796909106f20396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.615820 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:22.615764 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 52515.4us
I20251029 08:52:22.615724 2397 tablet.cc:1722] T 1da3e7940352460c9b2bb060174829c9 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:22.623732 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:22.623838 2156 raft_consensus.cc:493] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.623960 2156 raft_consensus.cc:515] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.624565 2156 leader_election.cc:290] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.625043 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.626220 1183 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.626451 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70b8c594aa24f37a169d1aa29c2268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.626658 1183 ts_tablet_manager.cc:1403] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.219s user 0.007s sys 0.000s
I20251029 08:52:22.628334 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_def435193a6b4fb893c46c0029059885" } new_table_name: "default.Table_21d20ee24eca44568b6a69044d5c55a4" modify_external_catalogs: true
I20251029 08:52:22.630501 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d64ea78b6d1648eabf348d4da0cffae8" } new_table_name: "default.Table_8d966fcf45004abdba1b04d1722be5c5" modify_external_catalogs: true
I20251029 08:52:22.630939 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ab7e172a704db3aa4c32da35993003. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.631376 2289 raft_consensus.cc:515] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.632211 1177 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.632344 1177 ts_tablet_manager.cc:1403] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.004s
I20251029 08:52:22.633955 1177 raft_consensus.cc:359] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.634120 1177 raft_consensus.cc:385] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.634177 1177 raft_consensus.cc:740] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.636191 2289 leader_election.cc:290] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.636914 1177 consensus_queue.cc:260] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.637916 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e65b3b1c8d14743aca1465d95ace95f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.640168 1177 ts_tablet_manager.cc:1434] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:22.640462 1177 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.641296 2289 raft_consensus.cc:493] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.642211 1177 tablet_bootstrap.cc:654] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.642206 2289 raft_consensus.cc:515] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.642858 2289 leader_election.cc:290] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.643003 2289 raft_consensus.cc:493] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.642921 2359 raft_consensus.cc:493] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.643110 2289 raft_consensus.cc:515] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.643116 2359 raft_consensus.cc:515] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.643612 2289 leader_election.cc:290] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.643662 2359 leader_election.cc:290] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.644173 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de78ea608c904aeea7f7145b9c80f45b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:22.644697 996 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.651815 1183 raft_consensus.cc:359] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.651983 1183 raft_consensus.cc:385] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.652060 1183 raft_consensus.cc:740] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.652287 1183 consensus_queue.cc:260] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.652712 1183 ts_tablet_manager.cc:1434] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251029 08:52:22.652781 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a186df0db24c4d22abee7ec57bcb199b
I20251029 08:52:22.653496 1183 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.654458 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 857fcd06914f42c1b94562915740da0d
I20251029 08:52:22.654912 1183 tablet_bootstrap.cc:654] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.655915 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b901ffd7ae47f487a355f08c076a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.656987 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet a186df0db24c4d22abee7ec57bcb199b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.658247 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f970b620cd40779691fb953e4dd520. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.663064 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:22.663977 1183 tablet_bootstrap.cc:492] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.664142 1183 ts_tablet_manager.cc:1403] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251029 08:52:22.664911 1183 raft_consensus.cc:359] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.665108 1183 raft_consensus.cc:385] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.665167 1183 raft_consensus.cc:740] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.665418 1183 consensus_queue.cc:260] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.665974 1183 ts_tablet_manager.cc:1434] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:22.666311 1183 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.667938 1183 tablet_bootstrap.cc:654] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.672150 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.672434 891 tablet_service.cc:1552] Processing DeleteTablet for tablet a186df0db24c4d22abee7ec57bcb199b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.672825 2156 raft_consensus.cc:493] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.672968 2156 raft_consensus.cc:515] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.673652 2156 leader_election.cc:290] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.674083 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ab9df6f768410eb4f66cffdcd68df2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:22.674609 861 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:22.675890 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a196d34a4b48988c07034dcc971049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.679558 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.683384 1176 ts_tablet_manager.cc:1929] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:22.683508 1176 log.cc:1199] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/afa2bb1075c4418eb10f4493f5e73c1b
I20251029 08:52:22.683856 1176 ts_tablet_manager.cc:1950] T afa2bb1075c4418eb10f4493f5e73c1b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:22.685829 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2ca9e5a740e943df8671c43f1e17dc05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.686396 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ca9e5a740e943df8671c43f1e17dc05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.688081 1179 ts_tablet_manager.cc:1929] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:22.688184 1179 log.cc:1199] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a3ba512bd4f04bd9aceb4cb210fe9689
I20251029 08:52:22.688381 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad98112c0ea4f59980e72165cef8a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.688544 1179 ts_tablet_manager.cc:1950] T a3ba512bd4f04bd9aceb4cb210fe9689 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:22.688999 2367 raft_consensus.cc:515] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:22.689522 1070 raft_consensus.cc:2393] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:22.689886 2367 leader_election.cc:290] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.690059 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_875a944d09b24fb98857e00795b8e64e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.690577 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_875a944d09b24fb98857e00795b8e64e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.694111 2366 raft_consensus.cc:515] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.697607 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1c023b679b4c3298bf953b53d4eb96" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:22.697850 922 raft_consensus.cc:2393] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:22.698944 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_180698a62af24093bee27beaedfc0c18" } modify_external_catalogs: true
I20251029 08:52:22.699151 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_180698a62af24093bee27beaedfc0c18" } modify_external_catalogs: true
I20251029 08:52:22.700594 2366 leader_election.cc:290] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.704432 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:22.704982 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_31cece98e77343e28a420add5c5d9c0e" } new_table_name: "default.Table_0b522a4d5a9949f79a2a0923a519ec77" modify_external_catalogs: true
I20251029 08:52:22.706332 2295 raft_consensus.cc:515] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.709326 922 raft_consensus.cc:2393] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:22.711154 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 895) took 1235 ms. Trace:
I20251029 08:52:22.711315 1176 rpcz_store.cc:276] 1029 08:52:21.475222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.478747 (+ 3525us) service_pool.cc:225] Handling call
1029 08:52:22.511396 (+1032649us) ts_tablet_manager.cc:1151] Deleting tablet afa2bb1075c4418eb10f4493f5e73c1b
1029 08:52:22.511406 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:22.680160 (+168754us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:22.683259 (+ 3099us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:22.711102 (+ 27843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032348}
I20251029 08:52:22.711660 1176 tablet_replica.cc:333] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:22.712280 1176 raft_consensus.cc:2243] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.716246 1176 raft_consensus.cc:2272] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.716486 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe384ca428974ac5ba78e6506a1a8832" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:22.716954 1176 ts_tablet_manager.cc:1916] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.717182 1070 raft_consensus.cc:2393] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:22.718219 1179 tablet_replica.cc:333] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:22.718405 1179 raft_consensus.cc:2243] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.718552 1179 raft_consensus.cc:2272] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.719041 1179 ts_tablet_manager.cc:1916] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.723227 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a3ba512bd4f04bd9aceb4cb210fe9689 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:22.731532 1179 ts_tablet_manager.cc:1929] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:22.736904 1177 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.737056 1177 ts_tablet_manager.cc:1403] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.000s
I20251029 08:52:22.740850 1177 raft_consensus.cc:359] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.741043 1177 raft_consensus.cc:385] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.741094 1177 raft_consensus.cc:740] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.741295 1177 consensus_queue.cc:260] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:22.750204 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f816afb0d084c6db44be8a41f53ce04 has no consensus state
I20251029 08:52:22.750234 1177 ts_tablet_manager.cc:1434] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251029 08:52:22.750542 1177 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.752813 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5c93076ca44542a5ae19e2c73dba95cb
I20251029 08:52:22.753589 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_61766253e4ff493b98bfa21f283633e7" } new_table_name: "default.Table_36fe2036cdd44511b7b4e1c042f913ee" modify_external_catalogs: true
I20251029 08:52:22.754552 1177 tablet_bootstrap.cc:654] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.757694 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0f0ca950566740a49562ebfa6fb1772a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.758253 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0ca950566740a49562ebfa6fb1772a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.759908 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c2392d03cd0b46e08f84a4eee1831855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.760492 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2392d03cd0b46e08f84a4eee1831855 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.763214 2295 leader_election.cc:290] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:22.766134 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c74b052ac94eae9e226285504c6b40 consensus state has no leader
I20251029 08:52:22.766480 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8364a95e36094945a4d55d737dc1e117" } new_table_name: "default.Table_ab4e50ed7bad43b4adab2ec2cf76a327" modify_external_catalogs: true
I20251029 08:52:22.768468 2295 raft_consensus.cc:493] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.768601 2295 raft_consensus.cc:515] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
W20251029 08:52:22.769928 861 leader_election.cc:343] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 1ae561a995c94fadb288b6395c55a1bf
I20251029 08:52:22.770066 2295 leader_election.cc:290] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.777539 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:22.778121 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87c98206d7fa409b8017abc72688db93 consensus state has no leader
I20251029 08:52:22.778295 1178 ts_tablet_manager.cc:1434] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.242s user 0.007s sys 0.000s
I20251029 08:52:22.778842 1178 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:22.778978 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0fb0c73465d84a63805ee4f2b426b6de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.779693 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fb0c73465d84a63805ee4f2b426b6de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.781509 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c385a4409927408eb2a8e027b4eeba95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.781984 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c385a4409927408eb2a8e027b4eeba95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.781016 597 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_b58caa6aed5542948934aeca3e67d32d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc] alter complete (version 3)
I20251029 08:52:22.784271 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da" } modify_external_catalogs: true
I20251029 08:52:22.784535 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da" } modify_external_catalogs: true
I20251029 08:52:22.783358 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c0ef8fb316bd46f8894cd22f35970a80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.785295 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0ef8fb316bd46f8894cd22f35970a80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.790899 1974 ts_tablet_manager.cc:1929] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:22.791070 1974 log.cc:1199] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a3ba512bd4f04bd9aceb4cb210fe9689
I20251029 08:52:22.791555 1974 ts_tablet_manager.cc:1950] T a3ba512bd4f04bd9aceb4cb210fe9689 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:22.792128 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a3ba512bd4f04bd9aceb4cb210fe9689 (table default.Table_3137cb7a7def4c5693f750cec7eb3f47 [id=85f838ceb1474854b675c525fbf1a590]) successfully deleted
I20251029 08:52:22.793409 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_8dce1780dcf74468bd3a72090ccf984c [id=3300362dc3c7436099416c59660b1e68] alter complete (version 1)
I20251029 08:52:22.794564 2295 raft_consensus.cc:493] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.794901 2295 raft_consensus.cc:515] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.795970 2295 leader_election.cc:290] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.798408 889 tablet_service.cc:1505] Processing CreateTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 (DEFAULT_TABLE table=default.Table_f1dca37ca0fc4bc38418e729d7fbff0b [id=b4d5a8e0baa547528e433b6ffdc023ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.798872 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bcc468ce964610a857bc2b11b5b3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.799341 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b71b3c0fa5504156ab80c2861bb9c67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.799876 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b71b3c0fa5504156ab80c2861bb9c67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.804643 887 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.804899 890 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:22.813336 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6691e1944ae947b49e4431f31ba9ae41 consensus state has no leader
I20251029 08:52:22.813807 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_871a2454944f4691b30571553cc9bbbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.814352 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_871a2454944f4691b30571553cc9bbbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.816707 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_74466c51bbac4f8a9c086680a8eda1d0" } modify_external_catalogs: true
I20251029 08:52:22.816852 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_74466c51bbac4f8a9c086680a8eda1d0" } modify_external_catalogs: true
W20251029 08:52:22.820539 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ed4f048f1ae4d70aa1b88116a57e2bd consensus state has no leader
I20251029 08:52:22.736057 1183 tablet_bootstrap.cc:492] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.824465 1179 log.cc:1199] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/351047c725f8488abbd074bd40790b48
I20251029 08:52:22.824548 1183 ts_tablet_manager.cc:1403] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.000s
I20251029 08:52:22.824855 1179 ts_tablet_manager.cc:1950] T 351047c725f8488abbd074bd40790b48 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:22.825400 1183 raft_consensus.cc:359] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.825596 1183 raft_consensus.cc:385] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.825663 1183 raft_consensus.cc:740] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.825873 1183 consensus_queue.cc:260] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.825938 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 351047c725f8488abbd074bd40790b48 (table default.Table_04d72f6fb117444a96f5d646217085e2 [id=de08f5f1950e4352a63c7d056bf4ce86]) successfully deleted
I20251029 08:52:22.826404 1183 ts_tablet_manager.cc:1434] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:22.826697 1183 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.756418 2397 tablet.cc:1722] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:22.758253 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.795868 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c66cbb495f5142d79a674f7d5adc505e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:22.830013 1067 raft_consensus.cc:2468] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:22.808490 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 (DEFAULT_TABLE table=default.Table_f1dca37ca0fc4bc38418e729d7fbff0b [id=b4d5a8e0baa547528e433b6ffdc023ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.810482 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 63bbad742c4b472fa42745b6550fbbbe (DEFAULT_TABLE table=default.Table_d25e6fa400a94d7ca4b9b5c566688235 [id=1709163424424b2081b676f0978049e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.830500 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bcc468ce964610a857bc2b11b5b3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.830964 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bbad742c4b472fa42745b6550fbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.831367 861 leader_election.cc:304] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.831946 1178 tablet_bootstrap.cc:654] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.832362 2295 raft_consensus.cc:2804] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.832456 2295 raft_consensus.cc:493] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.832518 2295 raft_consensus.cc:3060] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.835572 2366 raft_consensus.cc:493] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.835688 2366 raft_consensus.cc:515] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.836318 2366 leader_election.cc:290] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.836817 2295 raft_consensus.cc:515] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.838577 2295 leader_election.cc:290] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.840149 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c66cbb495f5142d79a674f7d5adc505e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:22.840314 1067 raft_consensus.cc:3060] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.840462 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aafa846cb2144cec9ce4b28e61d91961" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.840653 922 raft_consensus.cc:2468] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:22.843155 1067 raft_consensus.cc:2468] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:22.843573 861 leader_election.cc:304] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.843884 2295 raft_consensus.cc:2804] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:22.844944 1183 tablet_bootstrap.cc:654] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.848245 2295 raft_consensus.cc:697] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:22.848520 2295 consensus_queue.cc:237] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.849663 996 leader_election.cc:304] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.849913 2366 raft_consensus.cc:2804] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.849987 2366 raft_consensus.cc:493] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.850055 2366 raft_consensus.cc:3060] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.852684 2366 raft_consensus.cc:515] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.853433 2366 leader_election.cc:290] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.853649 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aafa846cb2144cec9ce4b28e61d91961" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:22.853842 922 raft_consensus.cc:3060] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.856750 922 raft_consensus.cc:2468] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:22.859968 996 leader_election.cc:304] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.860287 2367 raft_consensus.cc:2804] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:22.860345 2366 raft_consensus.cc:493] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.860402 2367 raft_consensus.cc:697] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:22.860453 2366 raft_consensus.cc:515] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.860615 2367 consensus_queue.cc:237] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.861104 2366 leader_election.cc:290] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.862890 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e1f0cdfbc34c6f85f2ad545eb33c0d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.863054 922 raft_consensus.cc:2468] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:22.863610 996 leader_election.cc:304] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.864073 1177 tablet_bootstrap.cc:492] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.864216 1177 ts_tablet_manager.cc:1403] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.114s user 0.019s sys 0.013s
I20251029 08:52:22.864216 2359 raft_consensus.cc:2804] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.864287 2359 raft_consensus.cc:493] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.864360 2359 raft_consensus.cc:3060] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.864493 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_36fe2036cdd44511b7b4e1c042f913ee" } modify_external_catalogs: true
I20251029 08:52:22.864706 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_36fe2036cdd44511b7b4e1c042f913ee" } modify_external_catalogs: true
I20251029 08:52:22.864964 1177 raft_consensus.cc:359] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.865111 1177 raft_consensus.cc:385] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.865159 1177 raft_consensus.cc:740] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.866817 1177 consensus_queue.cc:260] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.867113 2359 raft_consensus.cc:515] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.867899 2359 leader_election.cc:290] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:22.867317 1177 ts_tablet_manager.cc:1434] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:22.868570 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f95f49b0100b4f0f9fdb61e2ff742ac9" } modify_external_catalogs: true
I20251029 08:52:22.868764 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f95f49b0100b4f0f9fdb61e2ff742ac9" } modify_external_catalogs: true
I20251029 08:52:22.869208 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e2b87d133f8a4bd1877e07a1a8758147
I20251029 08:52:22.870698 1177 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.871791 890 tablet_service.cc:1552] Processing DeleteTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.872887 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e1f0cdfbc34c6f85f2ad545eb33c0d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:22.873055 922 raft_consensus.cc:3060] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.874822 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.874835 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.875902 922 raft_consensus.cc:2468] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:22.879504 1177 tablet_bootstrap.cc:654] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.880463 996 leader_election.cc:304] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.880738 2367 raft_consensus.cc:2804] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:22.880849 2367 raft_consensus.cc:697] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:22.881065 2367 consensus_queue.cc:237] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.881865 1183 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.882043 1183 ts_tablet_manager.cc:1403] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20251029 08:52:22.882887 1183 raft_consensus.cc:359] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.883189 1183 raft_consensus.cc:385] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.883301 1183 raft_consensus.cc:740] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.883579 1183 consensus_queue.cc:260] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.884169 1183 ts_tablet_manager.cc:1434] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:22.885042 889 tablet_service.cc:1505] Processing CreateTablet for tablet 63bbad742c4b472fa42745b6550fbbbe (DEFAULT_TABLE table=default.Table_d25e6fa400a94d7ca4b9b5c566688235 [id=1709163424424b2081b676f0978049e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.885488 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bbad742c4b472fa42745b6550fbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.885638 1177 tablet_bootstrap.cc:492] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.885777 1177 ts_tablet_manager.cc:1403] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251029 08:52:22.886770 1177 raft_consensus.cc:359] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.886945 1177 raft_consensus.cc:385] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.887008 1177 raft_consensus.cc:740] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.887246 1177 consensus_queue.cc:260] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.887769 1177 ts_tablet_manager.cc:1434] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:22.888067 1177 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.889690 1177 tablet_bootstrap.cc:654] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.892275 1183 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.893327 890 tablet_service.cc:1505] Processing CreateTablet for tablet a2c13db40d854269b13340aa6acef83e (DEFAULT_TABLE table=default.Table_c510c2d3e9184bcb92acb666b6bbfb84 [id=4d58b91b18124c8290e944166a752ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.893798 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c13db40d854269b13340aa6acef83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.893965 1183 tablet_bootstrap.cc:654] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.900998 1177 tablet_bootstrap.cc:492] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.901142 1177 ts_tablet_manager.cc:1403] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
W20251029 08:52:22.901536 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa348d6fa8c644bd8fe7b297a53c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa348d6fa8c644bd8fe7b297a53c1251 already in progress: creating tablet
I20251029 08:52:22.901922 1177 raft_consensus.cc:359] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.902091 1177 raft_consensus.cc:385] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.902150 1177 raft_consensus.cc:740] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.902333 1177 consensus_queue.cc:260] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:22.902387 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b7a6008b4e6243ccaf166a31f1a9f62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6008b4e6243ccaf166a31f1a9f62c already in progress: creating tablet
I20251029 08:52:22.902809 1177 ts_tablet_manager.cc:1434] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:22.903081 1177 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:22.903175 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 63e705a0b5264eb7a6988129edbf5970 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e705a0b5264eb7a6988129edbf5970 already in progress: creating tablet
I20251029 08:52:22.903461 1974 tablet_replica.cc:333] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:22.903692 1974 raft_consensus.cc:2243] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.903919 1974 raft_consensus.cc:2272] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.904542 1974 ts_tablet_manager.cc:1916] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.909091 1183 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.909240 1183 ts_tablet_manager.cc:1403] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20251029 08:52:22.910043 1183 raft_consensus.cc:359] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.910214 1183 raft_consensus.cc:385] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.910266 1183 raft_consensus.cc:740] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.910547 1183 consensus_queue.cc:260] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.911039 1183 ts_tablet_manager.cc:1434] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:22.911316 1183 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.913003 2397 tablet.cc:1722] T 2db77250e3ef489e8ab7dbcfcbb54385 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:22.913419 1183 tablet_bootstrap.cc:654] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.914048 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61b901ffd7ae47f487a355f08c076a22
I20251029 08:52:22.915023 1177 tablet_bootstrap.cc:654] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.922008 2359 raft_consensus.cc:493] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:22.922238 2359 raft_consensus.cc:515] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.923046 2359 leader_election.cc:290] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.924700 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:22.926841 1177 tablet_bootstrap.cc:492] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.926983 1177 ts_tablet_manager.cc:1403] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20251029 08:52:22.927754 1177 raft_consensus.cc:359] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.927937 1177 raft_consensus.cc:385] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.927990 1177 raft_consensus.cc:740] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.928196 1177 consensus_queue.cc:260] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.930049 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc04616301354cfe873c825a9f5d7ab6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:22.930222 920 raft_consensus.cc:2468] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:22.930902 1177 ts_tablet_manager.cc:1434] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:22.931269 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6691e1944ae947b49e4431f31ba9ae41
I20251029 08:52:22.931847 1183 tablet_bootstrap.cc:492] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.931986 1183 ts_tablet_manager.cc:1403] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251029 08:52:22.932731 1183 raft_consensus.cc:359] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.932914 1183 raft_consensus.cc:385] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.932976 1183 raft_consensus.cc:740] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.933166 1183 consensus_queue.cc:260] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.933674 1183 ts_tablet_manager.cc:1434] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:22.933943 1183 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.934079 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b901ffd7ae47f487a355f08c076a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.935606 1183 tablet_bootstrap.cc:654] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.940974 996 leader_election.cc:304] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:22.941263 2359 raft_consensus.cc:2804] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:22.941350 2359 raft_consensus.cc:493] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:22.941426 2359 raft_consensus.cc:3060] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.933009 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.944380 1177 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.946090 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c18e0e01fdc54b8dab97ec339d2aaa07
I20251029 08:52:22.946209 1177 tablet_bootstrap.cc:654] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.946653 2359 raft_consensus.cc:515] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:22.947502 2359 leader_election.cc:290] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:22.947744 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.949915 887 tablet_service.cc:1552] Processing DeleteTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.951831 2269 tablet.cc:1722] T 2db77250e3ef489e8ab7dbcfcbb54385 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:22.953313 1179 tablet_replica.cc:333] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:22.953608 1179 raft_consensus.cc:2243] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:22.953814 1179 raft_consensus.cc:2272] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:22.954381 1179 ts_tablet_manager.cc:1916] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:22.961272 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6056be95de0a4b34ab9dcf62b2968444" } modify_external_catalogs: true
I20251029 08:52:22.961510 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6056be95de0a4b34ab9dcf62b2968444" } modify_external_catalogs: true
I20251029 08:52:22.963188 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b777b26318974b9f92919063554e2100" } new_table_name: "default.Table_4a8284ff6d314bbc976db77d1cb42d29" modify_external_catalogs: true
I20251029 08:52:22.963490 1177 tablet_bootstrap.cc:492] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:22.963639 1177 ts_tablet_manager.cc:1403] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20251029 08:52:22.964214 1049 tablet_service.cc:1552] Processing DeleteTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:22.964463 1177 raft_consensus.cc:359] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.964644 1177 raft_consensus.cc:385] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.964716 1177 raft_consensus.cc:740] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.964879 1183 tablet_bootstrap.cc:492] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.965013 1183 ts_tablet_manager.cc:1403] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.003s
I20251029 08:52:22.964931 1177 consensus_queue.cc:260] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.965561 1177 ts_tablet_manager.cc:1434] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:22.965847 1177 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:22.965749 1183 raft_consensus.cc:359] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.965910 1183 raft_consensus.cc:385] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.965965 1183 raft_consensus.cc:740] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.966154 1183 consensus_queue.cc:260] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.967096 887 tablet_service.cc:1505] Processing CreateTablet for tablet 980e7125f59140c8876e4befc1becd78 (DEFAULT_TABLE table=default.Table_691d182612d74f709b0e5d106e4a5e11 [id=b80784c8a13c4a34a17b23c9c1757cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:22.967159 1183 ts_tablet_manager.cc:1434] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:22.967480 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980e7125f59140c8876e4befc1becd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:22.967626 1183 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.967698 1177 tablet_bootstrap.cc:654] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.970306 1183 tablet_bootstrap.cc:654] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.970579 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d25e6fa400a94d7ca4b9b5c566688235" } new_table_name: "default.Table_cdc154cbecfd42c9b846f3cc9b7d5112" modify_external_catalogs: true
I20251029 08:52:22.971808 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_448dfa8e59de435faa660bb3e9ee4cd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:22.972271 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5d3db711683243eabfce1d1543d63f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:22.972769 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3db711683243eabfce1d1543d63f9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:22.974006 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_448dfa8e59de435faa660bb3e9ee4cd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:22.979190 1183 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:22.979200 590 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_6f92afb6d07b43a3a12cc959d139105a [id=e095ee6c7ffc44019821263c022eee81] alter complete (version 2)
I20251029 08:52:22.979354 1183 ts_tablet_manager.cc:1403] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251029 08:52:22.979703 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4876937bd3244d43b4154fdd5eb81de7" } new_table_name: "default.Table_7ee2369d41744dd6a935286ad2dfdfdf" modify_external_catalogs: true
I20251029 08:52:22.980139 1183 raft_consensus.cc:359] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.980299 1183 raft_consensus.cc:385] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:22.980352 1183 raft_consensus.cc:740] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:22.980538 1183 consensus_queue.cc:260] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:22.981038 1183 ts_tablet_manager.cc:1434] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:22.981343 1183 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:22.983093 1183 tablet_bootstrap.cc:654] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:22.986495 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc04616301354cfe873c825a9f5d7ab6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:22.987056 922 raft_consensus.cc:3060] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:22.989032 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:22.990275 922 raft_consensus.cc:2468] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:22.992741 892 tablet_service.cc:1552] Processing DeleteTablet for tablet fc04616301354cfe873c825a9f5d7ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.992983 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.993129 892 tablet_service.cc:1552] Processing DeleteTablet for tablet de78ea608c904aeea7f7145b9c80f45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:08 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.992681 887 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:22.996460 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b901ffd7ae47f487a355f08c076a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.000473 996 leader_election.cc:304] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.000545 2418 raft_consensus.cc:493] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.000885 2359 raft_consensus.cc:2804] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.001121 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet eed66c3fa3f848df905a2a31fd553474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:23.001183 2359 raft_consensus.cc:697] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:23.001266 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:23.001390 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ab9df6f768410eb4f66cffdcd68df2 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e35d40b730a4a7883c324a6cc93b161) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:23.001588 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:23.001801 2359 consensus_queue.cc:237] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.003453 1049 tablet_service.cc:1505] Processing CreateTablet for tablet a2c13db40d854269b13340aa6acef83e (DEFAULT_TABLE table=default.Table_c510c2d3e9184bcb92acb666b6bbfb84 [id=4d58b91b18124c8290e944166a752ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.003901 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c13db40d854269b13340aa6acef83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.005364 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 62597189ecb047788b83788b514e9099
I20251029 08:52:23.008865 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 62597189ecb047788b83788b514e9099 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.009975 2418 raft_consensus.cc:515] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.010848 2418 leader_election.cc:290] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.010824 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 62597189ecb047788b83788b514e9099 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:23.011674 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 437146us
I20251029 08:52:23.012960 780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "772de9216f2647f8ae52f594df272427" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
W20251029 08:52:23.014739 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fad7549758649229ad771de926aaf15 consensus state has no leader
W20251029 08:52:23.016516 861 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:23.016645 861 leader_election.cc:304] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.016880 2295 raft_consensus.cc:2749] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.021854 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_50bc8d05fa8346ebaaa0c1a6a198d929" } new_table_name: "default.Table_ae84d579cd964855a5e44aa279312302" modify_external_catalogs: true
I20251029 08:52:23.028570 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c32c0cc51f75438f8786a435871e0c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.029260 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c32c0cc51f75438f8786a435871e0c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.031656 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ef2ebd3d2f9d4f03b009a7e13471cfa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.032301 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef2ebd3d2f9d4f03b009a7e13471cfa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.032289 1176 ts_tablet_manager.cc:1929] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:23.032490 1176 log.cc:1199] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/cdcc9a441c6a4d82a58262f4fdb3f00d
I20251029 08:52:23.032981 1176 ts_tablet_manager.cc:1950] T cdcc9a441c6a4d82a58262f4fdb3f00d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:23.034344 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_40f4c90bb63e4ebdbf7af5f31149e57e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.034869 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40f4c90bb63e4ebdbf7af5f31149e57e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.037533 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_49c30f52e0aa44519111145ec3d20c12" } new_table_name: "default.Table_54a3a1198f764ce1bd9af5fa5b3258ef" modify_external_catalogs: true
I20251029 08:52:23.040704 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e65b3b1c8d14743aca1465d95ace95f" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.041785 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c5f70a8f965d4999a64b666bffc151c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.042465 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5f70a8f965d4999a64b666bffc151c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.042685 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a355c68ecfb45ff8c396873d91268a8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.048426 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 930314c8f90f462990f93fab8f7790c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.048667 745 tablet_service.cc:1552] Processing DeleteTablet for tablet dbad8c4014b6423cb20e3e5c22ed69f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.048808 748 tablet_service.cc:1505] Processing CreateTablet for tablet b91221923dae416dbf046b555f390d99 (DEFAULT_TABLE table=default.Table_47a0b540f1834bd8bd754a77e504c6e5 [id=3cf39271e2354e568d033bb9b13b0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.049791 757 tablet_service.cc:1505] Processing CreateTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f (DEFAULT_TABLE table=default.Table_1938936fb1a5483c9d118a57d88d283b [id=a272ed18c02643d59796909106f20396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.050319 752 tablet_service.cc:1505] Processing CreateTablet for tablet 63bbad742c4b472fa42745b6550fbbbe (DEFAULT_TABLE table=default.Table_d25e6fa400a94d7ca4b9b5c566688235 [id=1709163424424b2081b676f0978049e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.053099 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:23.047302 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63bbad742c4b472fa42745b6550fbbbe consensus state has no leader
W20251029 08:52:23.059398 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (table default.Table_86364f11715846199b42645e6deeb2fb [id=7e89e19d5a384f8195c22592fe17e43d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2fb9ab76dcfe486e89bbe9391cab56f9 consensus state has no leader
I20251029 08:52:23.060827 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24834985a71349b9865840e9b5537d72" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.064002 887 tablet_service.cc:1505] Processing CreateTablet for tablet d709bfaee2de4d3c990e06b415bc4daf (DEFAULT_TABLE table=default.Table_1949c3fc4543485b88fb4788719aed4c [id=b8e7053b22544457ba272ee8f6786d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.065277 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709bfaee2de4d3c990e06b415bc4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.070405 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ccddc094f01348a5b25ebd260db6d8cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.071043 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccddc094f01348a5b25ebd260db6d8cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.072858 890 tablet_service.cc:1505] Processing CreateTablet for tablet f17002d67d13458ea3e32220eba9cb01 (DEFAULT_TABLE table=default.Table_5f659bad54be4c2390461e5b06833ee4 [id=bdcb90efc8214c7fb8b151697d7d1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.074189 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17002d67d13458ea3e32220eba9cb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.077103 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91221923dae416dbf046b555f390d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.077867 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad98112c0ea4f59980e72165cef8a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.080091 745 tablet_service.cc:1505] Processing CreateTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 (DEFAULT_TABLE table=default.Table_f3b73a492eca4e8fa1f1365cbc3518af [id=6b47cbaac80a40f68255258a91d17440]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.080539 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0f1cdfb1804d1c8d6220907795fc80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.084107 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c93076ca44542a5ae19e2c73dba95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.084165 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.084385 747 tablet_service.cc:1505] Processing CreateTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 (DEFAULT_TABLE table=default.Table_f1dca37ca0fc4bc38418e729d7fbff0b [id=b4d5a8e0baa547528e433b6ffdc023ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.084915 754 tablet_service.cc:1552] Processing DeleteTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.084878 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a186df0db24c4d22abee7ec57bcb199b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.085112 749 tablet_service.cc:1505] Processing CreateTablet for tablet 5c02b6343368482e88604ed320147762 (DEFAULT_TABLE table=default.Table_2f924d8dc47a422b99f57fbaea177d74 [id=7730a5bff86f40bcb2e5879b79419e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.089629 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ffc14da79c544d394a4d7fe49b465f9" } modify_external_catalogs: true
I20251029 08:52:23.089834 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8ffc14da79c544d394a4d7fe49b465f9" } modify_external_catalogs: true
I20251029 08:52:23.090755 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b901ffd7ae47f487a355f08c076a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.091117 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6691e1944ae947b49e4431f31ba9ae41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.091322 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 898) took 1518 ms. Trace:
I20251029 08:52:23.091460 1176 rpcz_store.cc:276] 1029 08:52:21.516194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.516260 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:22.711570 (+1195310us) ts_tablet_manager.cc:1151] Deleting tablet cdcc9a441c6a4d82a58262f4fdb3f00d
1029 08:52:22.711581 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:22.722608 (+ 11027us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:22.878124 (+155516us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.034319 (+156195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195067}
I20251029 08:52:23.091984 1176 tablet_replica.cc:333] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:23.092008 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 980e7125f59140c8876e4befc1becd78 (DEFAULT_TABLE table=default.Table_691d182612d74f709b0e5d106e4a5e11 [id=b80784c8a13c4a34a17b23c9c1757cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.092283 1176 raft_consensus.cc:2243] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.092478 1176 raft_consensus.cc:2272] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.092551 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980e7125f59140c8876e4befc1becd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.092900 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bbad742c4b472fa42745b6550fbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.093123 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c02b6343368482e88604ed320147762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.096210 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bcc468ce964610a857bc2b11b5b3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.096431 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c18e0e01fdc54b8dab97ec339d2aaa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.097144 745 tablet_service.cc:1505] Processing CreateTablet for tablet a2c13db40d854269b13340aa6acef83e (DEFAULT_TABLE table=default.Table_c510c2d3e9184bcb92acb666b6bbfb84 [id=4d58b91b18124c8290e944166a752ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.097540 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c13db40d854269b13340aa6acef83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.103201 861 leader_election.cc:304] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.103722 2295 raft_consensus.cc:2749] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:23.108304 861 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:23.108361 1178 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:23.108445 861 leader_election.cc:304] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.108541 1178 ts_tablet_manager.cc:1403] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.330s user 0.002s sys 0.002s
I20251029 08:52:23.108669 2295 raft_consensus.cc:2749] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.108620 1045 tablet_service.cc:1505] Processing CreateTablet for tablet f17002d67d13458ea3e32220eba9cb01 (DEFAULT_TABLE table=default.Table_5f659bad54be4c2390461e5b06833ee4 [id=bdcb90efc8214c7fb8b151697d7d1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.109133 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17002d67d13458ea3e32220eba9cb01. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:23.109455 861 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:23.109550 861 leader_election.cc:304] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.109884 2295 raft_consensus.cc:2749] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251029 08:52:23.110302 861 leader_election.cc:343] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 1ae561a995c94fadb288b6395c55a1bf
I20251029 08:52:23.110394 861 leader_election.cc:304] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.110698 2295 raft_consensus.cc:2749] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.110889 2366 raft_consensus.cc:493] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.111001 2366 raft_consensus.cc:515] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.111742 2366 leader_election.cc:290] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.118153 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dd62833c9d548adabbbf71856a4b181" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.118602 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ea52c1c186c8469b9a0a7bfba7654b5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.119252 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea52c1c186c8469b9a0a7bfba7654b5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:23.122236 996 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.123318 998 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fe384ca428974ac5ba78e6506a1a8832
W20251029 08:52:23.123822 998 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:23.123960 998 leader_election.cc:304] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:23.123565 749 tablet_service.cc:1505] Processing CreateTablet for tablet f17002d67d13458ea3e32220eba9cb01 (DEFAULT_TABLE table=default.Table_5f659bad54be4c2390461e5b06833ee4 [id=bdcb90efc8214c7fb8b151697d7d1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.123540 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 62597189ecb047788b83788b514e9099 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.124364 2366 raft_consensus.cc:2749] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.124617 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17002d67d13458ea3e32220eba9cb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.124838 752 tablet_service.cc:1505] Processing CreateTablet for tablet 980e7125f59140c8876e4befc1becd78 (DEFAULT_TABLE table=default.Table_691d182612d74f709b0e5d106e4a5e11 [id=b80784c8a13c4a34a17b23c9c1757cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.125278 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980e7125f59140c8876e4befc1becd78. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:23.127882 861 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c66cbb495f5142d79a674f7d5adc505e
W20251029 08:52:23.128343 861 leader_election.cc:343] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:23.128641 1178 raft_consensus.cc:359] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.129799 1178 raft_consensus.cc:385] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.130601 1178 raft_consensus.cc:740] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.137379 920 raft_consensus.cc:3060] T 772de9216f2647f8ae52f594df272427 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:23.137827 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d87191d05d4145a796a8823f723bc499 consensus state has no leader
I20251029 08:52:23.139006 1183 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:23.139173 1183 ts_tablet_manager.cc:1403] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.158s user 0.023s sys 0.025s
I20251029 08:52:23.140004 1183 raft_consensus.cc:359] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.140815 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a355c68ecfb45ff8c396873d91268a8" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.141139 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.142063 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet afa2bb1075c4418eb10f4493f5e73c1b (table default.Table_c719e48390fe4e1ca8e2ec1a6a4d168c [id=14d3bbf59230416e97563c8371a9fcd0]) successfully deleted
I20251029 08:52:23.145326 892 tablet_service.cc:1505] Processing CreateTablet for tablet d965d5a6b1324b00ad68b28d76796e80 (DEFAULT_TABLE table=default.Table_95ce9269797c4de3b5bd2c1cc2a6f384 [id=27424d0194344835af131cb4762ed2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.145776 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d965d5a6b1324b00ad68b28d76796e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.093127 1176 ts_tablet_manager.cc:1916] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.150104 896 tablet_service.cc:1505] Processing CreateTablet for tablet 1e0e85daedef44a1af1e4fc46ceeab65 (DEFAULT_TABLE table=default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.150547 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0e85daedef44a1af1e4fc46ceeab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.156893 902 tablet_service.cc:1505] Processing CreateTablet for tablet bb284e3ca1254697830c0208a9cb7574 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.157320 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb284e3ca1254697830c0208a9cb7574. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.158115 888 tablet_service.cc:1505] Processing CreateTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 (DEFAULT_TABLE table=default.Table_5e2a0f712be74b21b5ca4007fb38bf9d [id=e1f2336015f04e749f481e7b645681b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.158437 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bc2f2f5d6e418999fa40a6fadbbc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.172505 888 tablet_service.cc:1505] Processing CreateTablet for tablet 9e2554e46fb44660893e95a0654306e9 (DEFAULT_TABLE table=default.Table_6472d3c59db1490eb91b81a60372f0d4 [id=61f1666b91074d3aa1ec2005897da045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.173096 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2554e46fb44660893e95a0654306e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.174499 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet cdcc9a441c6a4d82a58262f4fdb3f00d (table default.Table_3e111656804b44da9e8317170d1f3605 [id=4d40ec0cb8ce4a46bf6a6083acdc1fde]) successfully deleted
I20251029 08:52:23.180876 892 tablet_service.cc:1505] Processing CreateTablet for tablet 7005bc082dc8400280525f5243660ecb (DEFAULT_TABLE table=default.Table_837fc07aec184f858821c9c37cb761cf [id=796e7e2be52d442abbe224c22ba8e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.181587 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7005bc082dc8400280525f5243660ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.183504 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24834985a71349b9865840e9b5537d72" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.184170 1183 raft_consensus.cc:385] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.184368 1183 raft_consensus.cc:740] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.184824 1183 consensus_queue.cc:260] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.186134 1183 ts_tablet_manager.cc:1434] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20251029 08:52:23.186683 1183 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:23.189934 2359 raft_consensus.cc:493] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.190184 2359 raft_consensus.cc:515] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.191097 2359 leader_election.cc:290] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.192050 1974 ts_tablet_manager.cc:1929] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.196703 1179 ts_tablet_manager.cc:1929] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.200137 1179 log.cc:1199] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:23.200728 1179 ts_tablet_manager.cc:1950] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:23.201355 2359 raft_consensus.cc:493] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.203208 2359 raft_consensus.cc:515] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.203974 2359 leader_election.cc:290] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.205044 1974 log.cc:1199] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/b7da7e08b74f4a9aacedada42e839a11
I20251029 08:52:23.205683 1974 ts_tablet_manager.cc:1950] T b7da7e08b74f4a9aacedada42e839a11 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:23.204793 1177 tablet_bootstrap.cc:492] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.194239 2295 raft_consensus.cc:493] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.206105 1177 ts_tablet_manager.cc:1403] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.240s user 0.014s sys 0.046s
I20251029 08:52:23.206130 2295 raft_consensus.cc:515] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.206833 1177 raft_consensus.cc:359] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.206919 2295 leader_election.cc:290] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.206982 1177 raft_consensus.cc:385] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.207029 1177 raft_consensus.cc:740] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.207206 1177 consensus_queue.cc:260] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.207707 1177 ts_tablet_manager.cc:1434] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:23.202610 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 882) took 1304 ms. Trace:
I20251029 08:52:23.207851 1179 rpcz_store.cc:276] 1029 08:52:21.897848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.897904 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:22.953258 (+1055354us) ts_tablet_manager.cc:1151] Deleting tablet 9e1c023b679b4c3298bf953b53d4eb96
1029 08:52:22.953269 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:22.957610 (+ 4341us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.196577 (+238967us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.202569 (+ 5992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055173}
I20251029 08:52:23.208266 1179 tablet_replica.cc:333] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:23.208456 1179 raft_consensus.cc:2243] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:23.208699 1179 pending_rounds.cc:70] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:23.208757 1179 pending_rounds.cc:77] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037654045618176 op_type: NO_OP noop_request { }
I20251029 08:52:23.208858 1179 raft_consensus.cc:2889] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:23.208952 1179 raft_consensus.cc:2272] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.215042 1183 tablet_bootstrap.cc:654] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.215098 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41e435e9dfb34d4f8b0c8f7b702d5a6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.215413 1067 raft_consensus.cc:2468] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.215950 861 leader_election.cc:304] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.216079 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:23.216313 2295 raft_consensus.cc:2804] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.216390 2295 raft_consensus.cc:493] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.216449 2295 raft_consensus.cc:3060] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.219161 2295 raft_consensus.cc:515] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.219827 2295 leader_election.cc:290] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.224735 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41e435e9dfb34d4f8b0c8f7b702d5a6c" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:23.225013 1067 raft_consensus.cc:3060] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.232748 1067 raft_consensus.cc:2468] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:23.236724 861 leader_election.cc:304] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.237598 2295 raft_consensus.cc:2804] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.237742 2156 raft_consensus.cc:493] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.237859 2156 raft_consensus.cc:515] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.238540 2156 leader_election.cc:290] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.240671 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.241529 1067 raft_consensus.cc:2468] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.242337 861 leader_election.cc:304] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.242594 2156 raft_consensus.cc:2804] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.242681 2156 raft_consensus.cc:493] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.242765 2156 raft_consensus.cc:3060] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.245862 2156 raft_consensus.cc:515] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.246749 2156 leader_election.cc:290] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.247844 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 927) took 1069 ms. Trace:
I20251029 08:52:23.247942 1974 rpcz_store.cc:276] 1029 08:52:22.178685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.178882 (+ 197us) service_pool.cc:225] Handling call
1029 08:52:22.903424 (+724542us) ts_tablet_manager.cc:1151] Deleting tablet b7da7e08b74f4a9aacedada42e839a11
1029 08:52:22.903432 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.134115 (+230683us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.191915 (+ 57800us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.247826 (+ 55911us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":724388}
I20251029 08:52:23.248221 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:23.248430 1067 raft_consensus.cc:3060] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.248435 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 928) took 1008 ms. Trace:
I20251029 08:52:23.248517 1974 rpcz_store.cc:276] 1029 08:52:22.239997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.240078 (+ 81us) service_pool.cc:225] Handling call
1029 08:52:23.248299 (+1008221us) ts_tablet_manager.cc:1151] Deleting tablet b582dfeab662456ba4730f5c80455d83
1029 08:52:23.248309 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.248415 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008027}
I20251029 08:52:23.249316 1974 tablet_replica.cc:333] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:23.249328 2359 raft_consensus.cc:493] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.249529 1974 raft_consensus.cc:2243] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.249485 2359 raft_consensus.cc:515] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.249702 1974 raft_consensus.cc:2272] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.250083 2359 leader_election.cc:290] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.250285 1974 ts_tablet_manager.cc:1916] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.250654 1177 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.251035 887 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.251484 1067 raft_consensus.cc:2468] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:23.251940 861 leader_election.cc:304] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.252192 889 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3782c226c14143aec56367d1a04996 (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.252629 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3782c226c14143aec56367d1a04996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.252652 1176 ts_tablet_manager.cc:1929] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.252810 1176 log.cc:1199] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/09968d93203140fa9201928c87702ee0
I20251029 08:52:23.253264 1176 ts_tablet_manager.cc:1950] T 09968d93203140fa9201928c87702ee0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:23.257473 1177 tablet_bootstrap.cc:654] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.258275 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.260149 2156 raft_consensus.cc:2804] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.260284 2156 raft_consensus.cc:697] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:23.260524 2156 consensus_queue.cc:237] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.265789 2295 raft_consensus.cc:697] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:23.266119 2295 consensus_queue.cc:237] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.270804 1183 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:23.271003 1183 ts_tablet_manager.cc:1403] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.003s
I20251029 08:52:23.271092 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet b7da7e08b74f4a9aacedada42e839a11 (table default.Table_5dad5c1ca58e4b7d90f9117a3c5a3510 [id=3e076a3173874cdfb6ddc5ee7a51f8dd]) successfully deleted
I20251029 08:52:23.271191 1974 ts_tablet_manager.cc:1929] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.271303 1974 log.cc:1199] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/1ae561a995c94fadb288b6395c55a1bf
W20251029 08:52:23.271692 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b582dfeab662456ba4730f5c80455d83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b582dfeab662456ba4730f5c80455d83 already in progress: creating tablet
W20251029 08:52:23.272208 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0b61c710897f43f798d88d92c4676d0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b61c710897f43f798d88d92c4676d0d already in progress: creating tablet
W20251029 08:52:23.272723 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9499d0099d1c4edc843fb8d29ca3b1b6 already in progress: creating tablet
W20251029 08:52:23.273324 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d09ed524e97c4c1bb3c4aaea876415ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d09ed524e97c4c1bb3c4aaea876415ea already in progress: creating tablet
W20251029 08:52:23.273762 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 36bea657aa9044f88e50bf4e749ff01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36bea657aa9044f88e50bf4e749ff01e already in progress: creating tablet
I20251029 08:52:23.274560 1183 raft_consensus.cc:359] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.274801 1183 raft_consensus.cc:385] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.274871 1183 raft_consensus.cc:740] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.275624 1183 consensus_queue.cc:260] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.276216 1183 ts_tablet_manager.cc:1434] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20251029 08:52:23.278630 1974 ts_tablet_manager.cc:1950] T 1ae561a995c94fadb288b6395c55a1bf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:23.279563 1974 tablet_replica.cc:333] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:23.280287 1974 raft_consensus.cc:2243] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:23.280812 1974 pending_rounds.cc:70] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:23.281143 1974 pending_rounds.cc:77] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037653911326720 op_type: NO_OP noop_request { }
I20251029 08:52:23.277908 887 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad240f5057a413180f88f3a88f92dbb (DEFAULT_TABLE table=default.Table_849532564a7c439cb059763db74ef2d2 [id=1e546294191b432193636c8cf9b43e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.277513 889 tablet_service.cc:1505] Processing CreateTablet for tablet 165a5a792a2d421c84fc2767c243f604 (DEFAULT_TABLE table=default.Table_84f0a37a39f64def888bada4afc4ccb2 [id=74c1fe0f0a29463e9bbf91f2972ed689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.277873 890 tablet_service.cc:1505] Processing CreateTablet for tablet 919e5494b84343639a2b9b81b05f1c2c (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.282411 1974 raft_consensus.cc:2889] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:23.282799 1974 raft_consensus.cc:2272] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.283895 1049 tablet_service.cc:1505] Processing CreateTablet for tablet d709bfaee2de4d3c990e06b415bc4daf (DEFAULT_TABLE table=default.Table_1949c3fc4543485b88fb4788719aed4c [id=b8e7053b22544457ba272ee8f6786d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.284126 1041 tablet_service.cc:1505] Processing CreateTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 (DEFAULT_TABLE table=default.Table_5e2a0f712be74b21b5ca4007fb38bf9d [id=e1f2336015f04e749f481e7b645681b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.284159 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 1e0e85daedef44a1af1e4fc46ceeab65 (DEFAULT_TABLE table=default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.283864 1045 tablet_service.cc:1505] Processing CreateTablet for tablet d965d5a6b1324b00ad68b28d76796e80 (DEFAULT_TABLE table=default.Table_95ce9269797c4de3b5bd2c1cc2a6f384 [id=27424d0194344835af131cb4762ed2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.284126 1047 tablet_service.cc:1505] Processing CreateTablet for tablet bb284e3ca1254697830c0208a9cb7574 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.285744 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 9e2554e46fb44660893e95a0654306e9 (DEFAULT_TABLE table=default.Table_6472d3c59db1490eb91b81a60372f0d4 [id=61f1666b91074d3aa1ec2005897da045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.285964 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 919e5494b84343639a2b9b81b05f1c2c (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.285744 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 7005bc082dc8400280525f5243660ecb (DEFAULT_TABLE table=default.Table_837fc07aec184f858821c9c37cb761cf [id=796e7e2be52d442abbe224c22ba8e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.286131 1032 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.286237 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 901) took 1478 ms. Trace:
I20251029 08:52:23.286438 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3782c226c14143aec56367d1a04996 (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.286336 1176 rpcz_store.cc:276] 1029 08:52:21.807226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.807287 (+ 61us) service_pool.cc:225] Handling call
1029 08:52:23.091889 (+1284602us) ts_tablet_manager.cc:1151] Deleting tablet 09968d93203140fa9201928c87702ee0
1029 08:52:23.091900 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.199156 (+107256us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.252459 (+ 53303us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.285520 (+ 33061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284405}
I20251029 08:52:23.286693 1176 tablet_replica.cc:333] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:23.286746 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad240f5057a413180f88f3a88f92dbb (DEFAULT_TABLE table=default.Table_849532564a7c439cb059763db74ef2d2 [id=1e546294191b432193636c8cf9b43e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.286875 1176 raft_consensus.cc:2243] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.287246 1176 raft_consensus.cc:2272] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.288540 1176 ts_tablet_manager.cc:1916] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.290460 1177 tablet_bootstrap.cc:492] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.290689 1177 ts_tablet_manager.cc:1403] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.005s
I20251029 08:52:23.290838 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 1ae561a995c94fadb288b6395c55a1bf (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:23.291549 1177 raft_consensus.cc:359] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.291625 2395 log.cc:826] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Log is configured to *not* fsync() on all Append() calls
I20251029 08:52:23.291747 1177 raft_consensus.cc:385] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.291932 1177 raft_consensus.cc:740] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.292030 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_09a0437aa25049d8acd5bdf172b6e934"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.292872 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09a0437aa25049d8acd5bdf172b6e934 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.292693 1177 consensus_queue.cc:260] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.293704 1177 ts_tablet_manager.cc:1434] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:23.294278 1177 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.130914 1178 consensus_queue.cc:260] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.297554 1183 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:23.298662 1178 ts_tablet_manager.cc:1434] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.190s user 0.001s sys 0.001s
I20251029 08:52:23.299134 1178 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:23.299899 1183 tablet_bootstrap.cc:654] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.301805 1178 tablet_bootstrap.cc:654] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.301235 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919e5494b84343639a2b9b81b05f1c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.324173 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad240f5057a413180f88f3a88f92dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.325224 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af38662420d14ac1a7cb5fd03fc1b9d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.325441 922 raft_consensus.cc:2468] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:23.327790 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9e1c023b679b4c3298bf953b53d4eb96 (table default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]) successfully deleted
I20251029 08:52:23.328449 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbc4345cabbc476485b6a040a076ddaf" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.328665 920 raft_consensus.cc:2468] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:23.329097 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2554e46fb44660893e95a0654306e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.330338 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ab9df6f768410eb4f66cffdcd68df2" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.330576 920 raft_consensus.cc:2468] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:23.332166 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3782c226c14143aec56367d1a04996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.336169 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7005bc082dc8400280525f5243660ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.340163 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.344177 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709bfaee2de4d3c990e06b415bc4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.346544 896 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.346897 1183 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:23.347046 1183 ts_tablet_manager.cc:1403] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.003s
I20251029 08:52:23.347860 1183 raft_consensus.cc:359] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.348083 1183 raft_consensus.cc:385] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.348167 1183 raft_consensus.cc:740] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.348433 1183 consensus_queue.cc:260] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.348176 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0e85daedef44a1af1e4fc46ceeab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.348958 1183 ts_tablet_manager.cc:1434] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:23.349643 1183 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:23.352175 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d965d5a6b1324b00ad68b28d76796e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.352456 1183 tablet_bootstrap.cc:654] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.354661 1974 ts_tablet_manager.cc:1916] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.356487 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb284e3ca1254697830c0208a9cb7574. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.360260 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bc2f2f5d6e418999fa40a6fadbbc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.370165 2359 raft_consensus.cc:493] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.370360 2359 raft_consensus.cc:515] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.371026 2359 leader_election.cc:290] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.375698 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 5 got 0
I20251029 08:52:23.382133 591 catalog_manager.cc:5649] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:23.382627 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:23.390323 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:23.374747 1176 ts_tablet_manager.cc:1929] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.390820 1176 log.cc:1199] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:23.391230 1176 ts_tablet_manager.cc:1950] T d3955ff79faf42bb82f764aae79b85d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:23.391615 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 902) took 1584 ms. Trace:
I20251029 08:52:23.391755 1176 rpcz_store.cc:276] 1029 08:52:21.807315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.807523 (+ 208us) service_pool.cc:225] Handling call
1029 08:52:23.286642 (+1479119us) ts_tablet_manager.cc:1151] Deleting tablet d3955ff79faf42bb82f764aae79b85d9
1029 08:52:23.286654 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.325307 (+ 38653us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.374622 (+ 49315us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.391597 (+ 16975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478976}
I20251029 08:52:23.392223 1176 tablet_replica.cc:333] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:23.392469 1176 raft_consensus.cc:2243] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.392747 1176 raft_consensus.cc:2272] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.393740 1176 ts_tablet_manager.cc:1916] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.402431 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet af38662420d14ac1a7cb5fd03fc1b9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:23.129655 720 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.405581 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 275938us
W20251029 08:52:23.405604 718 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.406024 720 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.406002 718 leader_election.cc:304] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:23.406160 720 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.406275 720 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.406316 718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 201141us
W20251029 08:52:23.406562 718 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.406672 718 leader_election.cc:304] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:23.407034 718 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.407119 718 leader_election.cc:304] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:23.302660 1179 ts_tablet_manager.cc:1916] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.407584 2422 raft_consensus.cc:2749] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.317710 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 165a5a792a2d421c84fc2767c243f604 (DEFAULT_TABLE table=default.Table_84f0a37a39f64def888bada4afc4ccb2 [id=74c1fe0f0a29463e9bbf91f2972ed689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.407852 2422 consensus_queue.cc:1048] T 772de9216f2647f8ae52f594df272427 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:23.407989 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:23.408104 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165a5a792a2d421c84fc2767c243f604. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.408155 784 raft_consensus.cc:2468] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.408253 2422 raft_consensus.cc:2749] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.408385 2422 raft_consensus.cc:2749] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.410359 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69d27944f3c14a6eb8331a904d6faa40" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:23.410550 784 raft_consensus.cc:3060] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.414351 784 raft_consensus.cc:2468] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:23.417814 861 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:23.420519 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919e5494b84343639a2b9b81b05f1c2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:23.421132 718 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.421380 718 leader_election.cc:304] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:23.421695 2422 raft_consensus.cc:2749] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.424152 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165a5a792a2d421c84fc2767c243f604. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.424357 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad240f5057a413180f88f3a88f92dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.427210 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 09968d93203140fa9201928c87702ee0 (table default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]) successfully deleted
I20251029 08:52:23.427687 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d3955ff79faf42bb82f764aae79b85d9 (table default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]) successfully deleted
I20251029 08:52:23.428550 2422 raft_consensus.cc:493] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.428706 2422 raft_consensus.cc:515] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.429407 2422 leader_election.cc:290] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:23.430567 752 tablet_service.cc:1505] Processing CreateTablet for tablet d965d5a6b1324b00ad68b28d76796e80 (DEFAULT_TABLE table=default.Table_95ce9269797c4de3b5bd2c1cc2a6f384 [id=27424d0194344835af131cb4762ed2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.430970 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d965d5a6b1324b00ad68b28d76796e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.431502 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9e2554e46fb44660893e95a0654306e9 (DEFAULT_TABLE table=default.Table_6472d3c59db1490eb91b81a60372f0d4 [id=61f1666b91074d3aa1ec2005897da045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.431627 758 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd (DEFAULT_TABLE table=default.Table_068eb03057e140249e1b348fd34910ce [id=1e29c1bd10ec4e769d740994df70231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.431845 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2554e46fb44660893e95a0654306e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.432046 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6ee4fe18d41bf8573dbd8f02f49dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.436272 1177 tablet_bootstrap.cc:654] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.436769 747 tablet_service.cc:1505] Processing CreateTablet for tablet 1e0e85daedef44a1af1e4fc46ceeab65 (DEFAULT_TABLE table=default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.437155 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0e85daedef44a1af1e4fc46ceeab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.438207 758 tablet_service.cc:1505] Processing CreateTablet for tablet 919e5494b84343639a2b9b81b05f1c2c (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.438583 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919e5494b84343639a2b9b81b05f1c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.440433 753 tablet_service.cc:1505] Processing CreateTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 (DEFAULT_TABLE table=default.Table_5e2a0f712be74b21b5ca4007fb38bf9d [id=e1f2336015f04e749f481e7b645681b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.440762 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bc2f2f5d6e418999fa40a6fadbbc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.443943 764 tablet_service.cc:1505] Processing CreateTablet for tablet 165a5a792a2d421c84fc2767c243f604 (DEFAULT_TABLE table=default.Table_84f0a37a39f64def888bada4afc4ccb2 [id=74c1fe0f0a29463e9bbf91f2972ed689]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.443912 757 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad240f5057a413180f88f3a88f92dbb (DEFAULT_TABLE table=default.Table_849532564a7c439cb059763db74ef2d2 [id=1e546294191b432193636c8cf9b43e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.444273 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165a5a792a2d421c84fc2767c243f604. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.444470 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad240f5057a413180f88f3a88f92dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.444285 746 tablet_service.cc:1505] Processing CreateTablet for tablet 7005bc082dc8400280525f5243660ecb (DEFAULT_TABLE table=default.Table_837fc07aec184f858821c9c37cb761cf [id=796e7e2be52d442abbe224c22ba8e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.444830 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7005bc082dc8400280525f5243660ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.430966 748 tablet_service.cc:1505] Processing CreateTablet for tablet bb284e3ca1254697830c0208a9cb7574 (DEFAULT_TABLE table=default.Table_2c83cedffb0e46e9be2088d13c2da515 [id=fda0cffdfda747fab2b2b2d066a75b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.445433 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb284e3ca1254697830c0208a9cb7574. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.446168 1177 tablet_bootstrap.cc:492] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.446292 1177 ts_tablet_manager.cc:1403] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.152s user 0.004s sys 0.000s
I20251029 08:52:23.446959 1177 raft_consensus.cc:359] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.447124 1177 raft_consensus.cc:385] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.447170 1177 raft_consensus.cc:740] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.447419 1177 consensus_queue.cc:260] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.447885 1177 ts_tablet_manager.cc:1434] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:23.448190 1177 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.430536 745 tablet_service.cc:1505] Processing CreateTablet for tablet d709bfaee2de4d3c990e06b415bc4daf (DEFAULT_TABLE table=default.Table_1949c3fc4543485b88fb4788719aed4c [id=b8e7053b22544457ba272ee8f6786d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.448268 754 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3782c226c14143aec56367d1a04996 (DEFAULT_TABLE table=default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:23.448577 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709bfaee2de4d3c990e06b415bc4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.452812 1177 tablet_bootstrap.cc:654] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.424142 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:23.456638 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.457032 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3782c226c14143aec56367d1a04996. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:23.457337 1179 ts_tablet_manager.cc:1929] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:23.458274 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d6d720fb6b54b1e8bb7b3956c48b3f9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.458616 1179 log.cc:1199] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/75e1f0cdfbc34c6f85f2ad545eb33c0d
I20251029 08:52:23.459129 1179 ts_tablet_manager.cc:1950] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:23.459726 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 883) took 1545 ms. Trace:
I20251029 08:52:23.461601 2359 raft_consensus.cc:493] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.461925 2359 raft_consensus.cc:515] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.462571 2359 leader_election.cc:290] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.462847 1179 rpcz_store.cc:276] 1029 08:52:21.914670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.914721 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:23.208194 (+1293473us) ts_tablet_manager.cc:1151] Deleting tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
1029 08:52:23.208204 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.443248 (+235044us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.457210 (+ 13962us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.459651 (+ 2441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293243}
I20251029 08:52:23.463640 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 884) took 1318 ms. Trace:
W20251029 08:52:23.463850 998 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:23.464437 998 leader_election.cc:304] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:23.465126 998 leader_election.cc:343] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:23.465267 998 leader_election.cc:304] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:23.463747 718 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:23.465449 998 raft_consensus.cc:2686] T 9e1c023b679b4c3298bf953b53d4eb96 P 730d0f5b7abf43a293b7df13d01ce4ee: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:23.465924 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5051ff776e4975b917db0393a1f945" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.466136 922 raft_consensus.cc:2468] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:23.466215 998 leader_election.cc:343] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:23.466346 998 leader_election.cc:304] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:23.466747 998 leader_election.cc:343] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aafa846cb2144cec9ce4b28e61d91961
W20251029 08:52:23.467031 998 leader_election.cc:343] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aafa846cb2144cec9ce4b28e61d91961
I20251029 08:52:23.467489 2359 raft_consensus.cc:2749] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.467687 2359 raft_consensus.cc:2749] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251029 08:52:23.468540 998 leader_election.cc:343] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 75e1f0cdfbc34c6f85f2ad545eb33c0d
W20251029 08:52:23.468927 998 leader_election.cc:343] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 75e1f0cdfbc34c6f85f2ad545eb33c0d
W20251029 08:52:23.469722 998 leader_election.cc:343] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fc04616301354cfe873c825a9f5d7ab6
W20251029 08:52:23.470091 998 leader_election.cc:343] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: fc04616301354cfe873c825a9f5d7ab6
I20251029 08:52:23.470234 1183 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:23.470489 1183 ts_tablet_manager.cc:1403] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.003s
W20251029 08:52:23.470538 998 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:23.470795 998 leader_election.cc:304] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:23.471114 2366 raft_consensus.cc:2749] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.472585 1177 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.472875 1177 ts_tablet_manager.cc:1403] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251029 08:52:23.473796 1177 raft_consensus.cc:359] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.474391 1177 raft_consensus.cc:385] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.474524 1177 raft_consensus.cc:740] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.474835 1177 consensus_queue.cc:260] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.475428 1177 ts_tablet_manager.cc:1434] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:23.475895 1177 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:23.478813 998 leader_election.cc:343] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:23.478835 1183 raft_consensus.cc:359] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.478104 1176 ts_tablet_manager.cc:1929] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.478751 1179 rpcz_store.cc:276] 1029 08:52:22.144806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.151777 (+ 6971us) service_pool.cc:225] Handling call
1029 08:52:23.463464 (+1311687us) ts_tablet_manager.cc:1151] Deleting tablet 0d08838d55e248ba99e00532bb82b65f
1029 08:52:23.463475 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.463584 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311402}
I20251029 08:52:23.479871 1176 log.cc:1199] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d5ac1af8b78a4d94b0e3da14ebfffd92
W20251029 08:52:23.479493 998 leader_election.cc:343] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:23.480805 2449 tablet.cc:1722] T 1da3e7940352460c9b2bb060174829c9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:23.481587 998 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:23.481994 996 leader_election.cc:304] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:23.482211 998 leader_election.cc:343] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7b5051ff776e4975b917db0393a1f945
I20251029 08:52:23.482592 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 885) took 1330 ms. Trace:
I20251029 08:52:23.484314 1179 rpcz_store.cc:276] 1029 08:52:22.152182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.152223 (+ 41us) service_pool.cc:225] Handling call
1029 08:52:23.482392 (+1330169us) ts_tablet_manager.cc:1151] Deleting tablet 82c522c0b20647b296d0439773a95aa6
1029 08:52:23.482406 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.482523 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1330021}
W20251029 08:52:23.482669 998 leader_election.cc:343] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:23.482999 2366 raft_consensus.cc:2804] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.485920 2366 raft_consensus.cc:493] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.486061 2366 raft_consensus.cc:3060] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.482594 1176 ts_tablet_manager.cc:1950] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:23.489068 2366 raft_consensus.cc:515] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.489811 2366 leader_election.cc:290] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:23.491498 998 leader_election.cc:343] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:23.483103 996 leader_election.cc:304] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.492324 2366 raft_consensus.cc:2804] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.492456 2366 raft_consensus.cc:493] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.492547 2366 raft_consensus.cc:3060] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.492815 996 leader_election.cc:304] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.493237 2359 raft_consensus.cc:2804] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.493368 2359 raft_consensus.cc:493] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.493494 2359 raft_consensus.cc:3060] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.495618 2366 raft_consensus.cc:515] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.496379 2366 leader_election.cc:290] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.496390 2359 raft_consensus.cc:515] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:23.497545 998 leader_election.cc:343] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:23.497741 2359 leader_election.cc:290] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.479496 1183 raft_consensus.cc:385] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.498272 1183 raft_consensus.cc:740] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:23.498456 998 leader_election.cc:343] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:23.477766 1177 tablet_bootstrap.cc:654] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.498766 1183 consensus_queue.cc:260] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.499992 1183 ts_tablet_manager.cc:1434] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.029s user 0.000s sys 0.004s
I20251029 08:52:23.503551 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 900) took 1696 ms. Trace:
I20251029 08:52:23.503612 2156 raft_consensus.cc:493] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.503777 2156 raft_consensus.cc:515] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.503896 2463 tablet.cc:1722] T 2db77250e3ef489e8ab7dbcfcbb54385 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:23.503786 1176 rpcz_store.cc:276] 1029 08:52:21.807119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.815278 (+ 8159us) service_pool.cc:225] Handling call
1029 08:52:23.392171 (+1576893us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:23.392181 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.422164 (+ 29983us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.476936 (+ 54772us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.503530 (+ 26594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564917}
I20251029 08:52:23.504673 996 leader_election.cc:304] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.504783 2156 leader_election.cc:290] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.505093 2359 raft_consensus.cc:2804] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.505262 2359 raft_consensus.cc:493] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.505411 2359 raft_consensus.cc:3060] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:23.508586 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef67d5cbc902492fb1252229b4b7f28f consensus state has no leader
I20251029 08:52:23.509016 2359 raft_consensus.cc:515] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.509519 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (table default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]) successfully deleted
I20251029 08:52:23.510398 2359 leader_election.cc:290] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.514544 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 903) took 1686 ms. Trace:
I20251029 08:52:23.514654 1176 rpcz_store.cc:276] 1029 08:52:21.827707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.828086 (+ 379us) service_pool.cc:225] Handling call
1029 08:52:23.504897 (+1676811us) ts_tablet_manager.cc:1151] Deleting tablet a4ae5978b71443a48a88bc69128b187d
1029 08:52:23.504906 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.514520 (+ 9614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676611}
W20251029 08:52:23.515175 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a4ae5978b71443a48a88bc69128b187d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4ae5978b71443a48a88bc69128b187d already in progress: creating tablet
W20251029 08:52:23.518430 998 leader_election.cc:343] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7b5051ff776e4975b917db0393a1f945
W20251029 08:52:23.524380 861 leader_election.cc:343] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:23.530571 1177 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.530910 1177 ts_tablet_manager.cc:1403] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.000s
I20251029 08:52:23.532397 1177 raft_consensus.cc:359] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.532788 1177 raft_consensus.cc:385] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.532925 1177 raft_consensus.cc:740] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.533389 1177 consensus_queue.cc:260] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.535279 1177 ts_tablet_manager.cc:1434] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:23.535925 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2b7853b85d24a04b8b11caaa4fe8c4d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.536221 1067 raft_consensus.cc:2468] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.537214 861 leader_election.cc:304] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.537634 2156 raft_consensus.cc:2804] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.537703 2156 raft_consensus.cc:493] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.537757 2156 raft_consensus.cc:3060] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.540635 2295 raft_consensus.cc:493] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.540827 2295 raft_consensus.cc:515] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.541019 2156 raft_consensus.cc:515] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.541462 2295 leader_election.cc:290] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.541855 2156 leader_election.cc:290] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.537071 1177 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.544083 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2b7853b85d24a04b8b11caaa4fe8c4d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:23.544083 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de78ea608c904aeea7f7145b9c80f45b" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.544397 1069 raft_consensus.cc:3060] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.544431 1070 raft_consensus.cc:2468] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.547613 1069 raft_consensus.cc:2468] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:23.551846 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 886) took 1399 ms. Trace:
I20251029 08:52:23.551986 1179 rpcz_store.cc:276] 1029 08:52:22.152502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.152541 (+ 39us) service_pool.cc:225] Handling call
1029 08:52:23.485765 (+1333224us) ts_tablet_manager.cc:1151] Deleting tablet 13afd85e234643e08957e7a7f7fd7e22
1029 08:52:23.551602 (+ 65837us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.551767 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333098}
I20251029 08:52:23.552551 1179 tablet_replica.cc:333] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:23.552781 1179 raft_consensus.cc:2243] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.553009 1179 raft_consensus.cc:2272] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.553036 1177 tablet_bootstrap.cc:654] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.553680 1179 ts_tablet_manager.cc:1916] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.564440 1183 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:23.566892 1183 tablet_bootstrap.cc:654] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.574091 1177 tablet_bootstrap.cc:492] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.574347 1177 ts_tablet_manager.cc:1403] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.004s
I20251029 08:52:23.575593 1177 raft_consensus.cc:359] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.575891 1177 raft_consensus.cc:385] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.575970 1177 raft_consensus.cc:740] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.576362 1177 consensus_queue.cc:260] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.577312 1177 ts_tablet_manager.cc:1434] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:23.579288 1177 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.582684 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eed66c3fa3f848df905a2a31fd553474" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.582921 922 raft_consensus.cc:2468] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:23.583592 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (table default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]) successfully deleted
I20251029 08:52:23.586490 1179 ts_tablet_manager.cc:1929] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.586803 1179 log.cc:1199] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:23.587709 1179 ts_tablet_manager.cc:1950] T fe384ca428974ac5ba78e6506a1a8832 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:23.588790 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 887) took 1436 ms. Trace:
W20251029 08:52:23.589524 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0d08838d55e248ba99e00532bb82b65f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d08838d55e248ba99e00532bb82b65f already in progress: creating tablet
I20251029 08:52:23.590031 861 leader_election.cc:304] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:23.590065 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 82c522c0b20647b296d0439773a95aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c522c0b20647b296d0439773a95aa6 already in progress: creating tablet
I20251029 08:52:23.591063 1177 tablet_bootstrap.cc:654] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.591420 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbc4345cabbc476485b6a040a076ddaf" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:23.590987 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af38662420d14ac1a7cb5fd03fc1b9d8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:23.591543 924 raft_consensus.cc:3060] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.591605 922 raft_consensus.cc:3060] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.591202 861 leader_election.cc:304] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.592593 2156 raft_consensus.cc:2804] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.592712 2156 raft_consensus.cc:697] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:23.592947 2156 consensus_queue.cc:237] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.594357 922 raft_consensus.cc:2468] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:23.590893 2468 raft_consensus.cc:2804] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.595278 2468 raft_consensus.cc:493] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.595427 2468 raft_consensus.cc:3060] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.595858 924 raft_consensus.cc:2468] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:23.599020 2468 raft_consensus.cc:515] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:23.605667 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 13afd85e234643e08957e7a7f7fd7e22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13afd85e234643e08957e7a7f7fd7e22 already in progress: creating tablet
I20251029 08:52:23.613078 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 904) took 1749 ms. Trace:
I20251029 08:52:23.613204 1176 rpcz_store.cc:276] 1029 08:52:21.863679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.864896 (+ 1217us) service_pool.cc:225] Handling call
1029 08:52:23.516291 (+1651395us) ts_tablet_manager.cc:1151] Deleting tablet dbf9e36c47be4509a998be8e7138475b
1029 08:52:23.612945 (+ 96654us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.613054 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651185}
I20251029 08:52:23.613588 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 905) took 1633 ms. Trace:
I20251029 08:52:23.613682 1176 rpcz_store.cc:276] 1029 08:52:21.980489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:21.980548 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:23.613482 (+1632934us) ts_tablet_manager.cc:1151] Deleting tablet 0d08838d55e248ba99e00532bb82b65f
1029 08:52:23.613494 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.613573 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632706}
I20251029 08:52:23.613976 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 906) took 1613 ms. Trace:
I20251029 08:52:23.614034 1176 rpcz_store.cc:276] 1029 08:52:22.000831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.001028 (+ 197us) service_pool.cc:225] Handling call
1029 08:52:23.613891 (+1612863us) ts_tablet_manager.cc:1151] Deleting tablet 82c522c0b20647b296d0439773a95aa6
1029 08:52:23.613898 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.613962 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612649}
I20251029 08:52:23.614290 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 907) took 1613 ms. Trace:
I20251029 08:52:23.614352 1176 rpcz_store.cc:276] 1029 08:52:22.000957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.001034 (+ 77us) service_pool.cc:225] Handling call
1029 08:52:23.614210 (+1613176us) ts_tablet_manager.cc:1151] Deleting tablet 13afd85e234643e08957e7a7f7fd7e22
1029 08:52:23.614217 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.614278 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612855}
I20251029 08:52:23.614559 1176 tablet_replica.cc:333] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:23.614734 1176 raft_consensus.cc:2243] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.614869 1176 raft_consensus.cc:2272] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.615353 1176 ts_tablet_manager.cc:1916] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:23.628780 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet dbf9e36c47be4509a998be8e7138475b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf9e36c47be4509a998be8e7138475b already in progress: creating tablet
W20251029 08:52:23.629262 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0d08838d55e248ba99e00532bb82b65f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d08838d55e248ba99e00532bb82b65f already in progress: creating tablet
W20251029 08:52:23.629563 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 82c522c0b20647b296d0439773a95aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c522c0b20647b296d0439773a95aa6 already in progress: creating tablet
W20251029 08:52:23.630817 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 13afd85e234643e08957e7a7f7fd7e22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13afd85e234643e08957e7a7f7fd7e22 already in progress: creating tablet
I20251029 08:52:23.632051 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5e2a0f712be74b21b5ca4007fb38bf9d" } modify_external_catalogs: true
I20251029 08:52:23.632256 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5e2a0f712be74b21b5ca4007fb38bf9d" } modify_external_catalogs: true
I20251029 08:52:23.633145 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 5 got 0
I20251029 08:52:23.633893 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4a5189be4b234b599b25a1f049a1b849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.634928 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a5189be4b234b599b25a1f049a1b849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.649421 1176 ts_tablet_manager.cc:1929] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:23.649566 1176 log.cc:1199] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:23.649910 1176 ts_tablet_manager.cc:1950] T 19fdc381a0d949b0b93cef399daa3021 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:23.650807 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 908) took 1648 ms. Trace:
I20251029 08:52:23.650894 1176 rpcz_store.cc:276] 1029 08:52:22.001957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.002092 (+ 135us) service_pool.cc:225] Handling call
1029 08:52:23.614522 (+1612430us) ts_tablet_manager.cc:1151] Deleting tablet 19fdc381a0d949b0b93cef399daa3021
1029 08:52:23.614529 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.626194 (+ 11665us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.649286 (+ 23092us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.650781 (+ 1495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612259}
I20251029 08:52:23.651461 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 19fdc381a0d949b0b93cef399daa3021 (table default.Table_17a9174cc94c45f1b499e06db7630b6f [id=033c1c3305724f0f9b37fd9729616d46]) successfully deleted
I20251029 08:52:23.663166 2422 raft_consensus.cc:493] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.663353 2422 raft_consensus.cc:515] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.666585 1178 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:23.666749 1178 ts_tablet_manager.cc:1403] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.368s user 0.001s sys 0.005s
I20251029 08:52:23.667474 1178 raft_consensus.cc:359] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.667721 1178 raft_consensus.cc:385] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.667770 1178 raft_consensus.cc:740] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.668561 2422 leader_election.cc:290] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:23.682688 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36c42e3bf3814769845a0204a00cf45f consensus state has no leader
I20251029 08:52:23.605194 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d6d720fb6b54b1e8bb7b3956c48b3f9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.684221 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9493653c2e354458923e9e8dd0da0c60
I20251029 08:52:23.684212 1179 rpcz_store.cc:276] 1029 08:52:22.152759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.152794 (+ 35us) service_pool.cc:225] Handling call
1029 08:52:23.552457 (+1399663us) ts_tablet_manager.cc:1151] Deleting tablet fe384ca428974ac5ba78e6506a1a8832
1029 08:52:23.552470 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.558342 (+ 5872us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.586177 (+ 27835us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.588737 (+ 2560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1399547}
I20251029 08:52:23.684667 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 896) took 1529 ms. Trace:
I20251029 08:52:23.665669 996 leader_election.cc:304] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.685492 996 leader_election.cc:304] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.686245 1176 tablet_replica.cc:333] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:23.686570 1176 raft_consensus.cc:2243] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.686811 1176 raft_consensus.cc:2272] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.687455 1176 ts_tablet_manager.cc:1916] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.688608 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:23.689323 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4" } modify_external_catalogs: true
I20251029 08:52:23.689497 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f7bf1bae501b4f6b8a65fe215a3f0ac4" } modify_external_catalogs: true
I20251029 08:52:23.690955 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8c17839d47d445acb4e7f4d30d5850dc" } modify_external_catalogs: true
I20251029 08:52:23.691141 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8c17839d47d445acb4e7f4d30d5850dc" } modify_external_catalogs: true
I20251029 08:52:23.692117 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4e81a355a1424094aa3b482cce58897a" } modify_external_catalogs: true
I20251029 08:52:23.692283 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4e81a355a1424094aa3b482cce58897a" } modify_external_catalogs: true
I20251029 08:52:23.692437 2359 raft_consensus.cc:2804] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.692476 2366 raft_consensus.cc:2804] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.692571 2359 raft_consensus.cc:493] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.692719 2359 raft_consensus.cc:3060] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.695307 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8c5b5a48612f4ed4a391c6cd25320eea" } new_table_name: "default.Table_4757cb1c3548493485ab674b57397de5" modify_external_catalogs: true
I20251029 08:52:23.695395 2359 raft_consensus.cc:515] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.696074 1179 rpcz_store.cc:276] 1029 08:52:22.155006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.155342 (+ 336us) service_pool.cc:225] Handling call
1029 08:52:23.684525 (+1529183us) ts_tablet_manager.cc:1151] Deleting tablet b7a6008b4e6243ccaf166a31f1a9f62c
1029 08:52:23.684537 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.684632 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529008}
I20251029 08:52:23.696933 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f3ebeac47874490abc642f878752012d" } new_table_name: "default.Table_5311ff040c54422a9359fdb7eb616ce7" modify_external_catalogs: true
I20251029 08:52:23.697031 2359 leader_election.cc:290] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.698412 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d03243f8356b4f47b90023ee624bbb95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.698959 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d03243f8356b4f47b90023ee624bbb95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:23.698997 998 leader_election.cc:343] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:23.699198 996 leader_election.cc:304] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:23.700577 720 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.700699 720 leader_election.cc:304] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:23.701097 2422 raft_consensus.cc:2749] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:23.702214 1183 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:23.702381 1183 ts_tablet_manager.cc:1403] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.138s user 0.001s sys 0.003s
I20251029 08:52:23.703294 1183 raft_consensus.cc:359] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.703480 1183 raft_consensus.cc:385] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.703583 1183 raft_consensus.cc:740] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.703858 1183 consensus_queue.cc:260] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.704303 2359 raft_consensus.cc:2804] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.704658 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 9493653c2e354458923e9e8dd0da0c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.705407 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet fe384ca428974ac5ba78e6506a1a8832 (table default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]) successfully deleted
I20251029 08:52:23.668007 1178 consensus_queue.cc:260] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.712284 2366 raft_consensus.cc:697] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:23.712935 2366 consensus_queue.cc:237] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.725080 1178 ts_tablet_manager.cc:1434] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.058s user 0.002s sys 0.000s
I20251029 08:52:23.728315 1178 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:23.600304 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b5051ff776e4975b917db0393a1f945" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:23.609427 1177 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.614331 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ab9df6f768410eb4f66cffdcd68df2" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:23.619495 861 leader_election.cc:343] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:23.738169 922 raft_consensus.cc:3060] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.738263 1177 ts_tablet_manager.cc:1403] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.004s
W20251029 08:52:23.738384 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 118897us
W20251029 08:52:23.738906 861 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:23.739181 1177 raft_consensus.cc:359] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.739391 1177 raft_consensus.cc:385] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.739478 1177 raft_consensus.cc:740] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.739530 2156 raft_consensus.cc:493] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.739660 2156 raft_consensus.cc:515] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.739727 1177 consensus_queue.cc:260] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.740276 2156 leader_election.cc:290] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:23.739193 861 leader_election.cc:343] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:23.741024 922 raft_consensus.cc:2468] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:23.669473 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:23.741393 919 raft_consensus.cc:2468] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:23.738195 924 raft_consensus.cc:3060] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.741590 1070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.742106 718 leader_election.cc:304] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.742508 1177 ts_tablet_manager.cc:1434] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:23.742830 1177 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.743054 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cd2e9a95b6c40be9e17397a313d249b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:23.743842 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9830b3c1094348f59df346203a20a135"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:23.744578 1177 tablet_bootstrap.cc:654] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.744529 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:23.744735 778 raft_consensus.cc:2468] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.745074 2422 raft_consensus.cc:2804] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.745151 2422 raft_consensus.cc:493] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.745231 2422 raft_consensus.cc:3060] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.745527 861 leader_election.cc:304] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:23.745831 861 leader_election.cc:343] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.746202 2451 raft_consensus.cc:2804] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.746302 2451 raft_consensus.cc:493] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.746404 2451 raft_consensus.cc:3060] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:23.748371 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9830b3c1094348f59df346203a20a135 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.749058 2451 raft_consensus.cc:515] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.749579 2451 leader_election.cc:290] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.751565 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:23.752310 861 leader_election.cc:343] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.752681 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:23.752880 778 raft_consensus.cc:3060] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.747937 2422 raft_consensus.cc:515] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.765312 2422 leader_election.cc:290] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.765717 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:23.765971 922 raft_consensus.cc:3060] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.770174 922 raft_consensus.cc:2468] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:23.772701 778 raft_consensus.cc:2468] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:23.686545 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 9493653c2e354458923e9e8dd0da0c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.703191 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eed66c3fa3f848df905a2a31fd553474" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:23.783738 920 raft_consensus.cc:3060] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.784231 718 leader_election.cc:304] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:23.784523 2422 raft_consensus.cc:2804] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.784722 2422 raft_consensus.cc:697] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:23.785604 2422 consensus_queue.cc:237] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.786600 920 raft_consensus.cc:2468] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:23.773279 861 leader_election.cc:304] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee
I20251029 08:52:23.787086 2451 raft_consensus.cc:2804] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.787237 2451 raft_consensus.cc:697] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:23.787505 2451 consensus_queue.cc:237] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.683821 1974 ts_tablet_manager.cc:1929] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:23.790315 1974 log.cc:1199] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:23.790957 1974 ts_tablet_manager.cc:1950] T c66cbb495f5142d79a674f7d5adc505e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:23.791465 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 940) took 1354 ms. Trace:
I20251029 08:52:23.791653 1974 rpcz_store.cc:276] 1029 08:52:22.437242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.437303 (+ 61us) service_pool.cc:225] Handling call
1029 08:52:23.279518 (+842215us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:23.279528 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.537986 (+258458us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.683703 (+145717us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:23.791441 (+107738us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":842040}
I20251029 08:52:23.792917 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c66cbb495f5142d79a674f7d5adc505e (table default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]) successfully deleted
I20251029 08:52:23.796054 2468 leader_election.cc:290] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.796898 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de78ea608c904aeea7f7145b9c80f45b" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:23.797112 1067 raft_consensus.cc:3060] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.797915 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.798213 1069 raft_consensus.cc:2468] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:23.800595 996 leader_election.cc:304] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.801342 996 leader_election.cc:304] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:23.802162 861 leader_election.cc:343] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:23.810527 2451 raft_consensus.cc:493] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.810892 2451 raft_consensus.cc:515] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.813705 2451 leader_election.cc:290] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:23.822527 861 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:23.829246 1974 tablet_replica.cc:333] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:23.829584 1974 raft_consensus.cc:2243] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:23.829869 1974 raft_consensus.cc:2272] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:23.830698 1974 ts_tablet_manager.cc:1916] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:23.831776 2458 raft_consensus.cc:493] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:23.831924 2458 raft_consensus.cc:515] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.832759 2458 leader_election.cc:290] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:23.846200 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b87d133f8a4bd1877e07a1a8758147" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:23.847258 718 leader_election.cc:343] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:23.801553 2474 raft_consensus.cc:2804] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.852424 2474 raft_consensus.cc:697] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:23.852746 2474 consensus_queue.cc:237] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.802559 1067 raft_consensus.cc:2468] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:23.812762 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dd62833c9d548adabbbf71856a4b181" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:23.854282 1069 raft_consensus.cc:2468] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:23.854878 861 leader_election.cc:304] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.855383 861 leader_election.cc:304] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:23.852092 2359 raft_consensus.cc:697] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:23.855758 2451 raft_consensus.cc:2804] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.855875 2451 raft_consensus.cc:697] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:23.855813 2359 consensus_queue.cc:237] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:23.856141 2451 consensus_queue.cc:237] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.857153 2295 raft_consensus.cc:2804] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:23.857230 2295 raft_consensus.cc:493] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:23.857297 2295 raft_consensus.cc:3060] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:23.860510 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 59195.3us
I20251029 08:52:23.861156 2475 raft_consensus.cc:2804] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:23.861338 2475 raft_consensus.cc:697] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:23.861725 2475 consensus_queue.cc:237] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.863422 1183 ts_tablet_manager.cc:1434] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.161s user 0.005s sys 0.000s
I20251029 08:52:23.864060 2295 raft_consensus.cc:515] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.864272 1183 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:23.864789 2295 leader_election.cc:290] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:23.866374 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dd62833c9d548adabbbf71856a4b181" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:23.866551 1069 raft_consensus.cc:3060] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:23.866982 1183 tablet_bootstrap.cc:654] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.868252 1178 tablet_bootstrap.cc:700] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent opening tablet: real 0.138s user 0.002s sys 0.000s
I20251029 08:52:23.868530 1178 tablet_bootstrap.cc:654] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:23.869185 861 leader_election.cc:343] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181
W20251029 08:52:23.869585 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 56895.6us
I20251029 08:52:23.886873 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d5bc2f2f5d6e418999fa40a6fadbbc71
I20251029 08:52:23.887882 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a92cc4a93a584dffaafe21316fe8dc06
I20251029 08:52:23.888487 590 master_service.cc:954] ReplaceTablet: received request to replace tablet b0b3b6fa45044c3fb83823ed4f233735 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:23.892637 758 tablet_service.cc:1552] Processing DeleteTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.897786 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.898083 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b5051ff776e4975b917db0393a1f945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.898250 889 tablet_service.cc:1552] Processing DeleteTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:23.905682 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:23.909580 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet fb22af7515084e099b41f9c1c92c067a
I20251029 08:52:23.910661 889 tablet_service.cc:1552] Processing DeleteTablet for tablet fb22af7515084e099b41f9c1c92c067a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:23.912034 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 527a83465a844849811b0f9d26428af0 consensus state has no leader
I20251029 08:52:23.913691 754 tablet_service.cc:1552] Processing DeleteTablet for tablet fb22af7515084e099b41f9c1c92c067a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:23.915990 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8052ae8345ad4f3f95714d7b9173d3d8 consensus state has no leader
I20251029 08:52:23.918919 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_480cdda5b5e84032a978c7065e456721" } new_table_name: "default.Table_af732cb4b90d49819386fe8dbb50ce73" modify_external_catalogs: true
I20251029 08:52:23.920444 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8cfe61c96c824cf4adcc750f24156238" } new_table_name: "default.Table_16e768ada8754f51a74b4d88207a0ddf" modify_external_catalogs: true
I20251029 08:52:23.920922 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9b4e0264c9d14e7cb206bd0bafa0a120"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.921438 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b4e0264c9d14e7cb206bd0bafa0a120 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.922312 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c097e84e85ba449185c70f18e0b3184c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.923563 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c097e84e85ba449185c70f18e0b3184c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.923815 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bc1f6a8467ac4e1084471956749c7145" } new_table_name: "default.Table_2b24b34725b34a2189255d1a59532f6b" modify_external_catalogs: true
I20251029 08:52:23.928650 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_45635496357a40e4ba3c778e9ca8b7b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.929314 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45635496357a40e4ba3c778e9ca8b7b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.932435 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2471fce4e7254548909d80a8b0c78ea9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:23.936537 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2471fce4e7254548909d80a8b0c78ea9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:23.946476 1177 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.946779 1177 ts_tablet_manager.cc:1403] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.204s user 0.026s sys 0.032s
I20251029 08:52:23.948206 1177 raft_consensus.cc:359] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.948480 1177 raft_consensus.cc:385] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.948556 1177 raft_consensus.cc:740] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.948930 1177 consensus_queue.cc:260] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.950912 1177 ts_tablet_manager.cc:1434] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:23.951990 1177 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.958218 1177 tablet_bootstrap.cc:654] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:23.967971 1177 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:23.968592 1177 ts_tablet_manager.cc:1403] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20251029 08:52:23.971601 1177 raft_consensus.cc:359] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.972330 1177 raft_consensus.cc:385] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:23.972539 1177 raft_consensus.cc:740] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:23.973472 1177 consensus_queue.cc:260] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:23.975697 1177 ts_tablet_manager.cc:1434] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251029 08:52:23.976842 1177 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:23.985417 1177 tablet_bootstrap.cc:654] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:23.992967 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33c7e63019634c229d824b2287683151 has no consensus state
W20251029 08:52:23.993571 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91622c8b93ac40679e1fea797838ea99 has no consensus state
I20251029 08:52:24.002485 593 master_service.cc:954] ReplaceTablet: received request to replace tablet 5f67e7b28b4240618e1817bb47d5633d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:24.002708 1177 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.003364 1177 ts_tablet_manager.cc:1403] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.001s
I20251029 08:52:24.003888 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_67c25e495c4d4292a9b38ecba68278e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.004644 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67c25e495c4d4292a9b38ecba68278e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.007467 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0d1d1cca556e4e6c91159f6cbbf6a2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.006996 1177 raft_consensus.cc:359] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:24.007987 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d1d1cca556e4e6c91159f6cbbf6a2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.007869 1177 raft_consensus.cc:385] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.008276 1177 raft_consensus.cc:740] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.009352 1069 raft_consensus.cc:2468] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:24.009492 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4d8e69bc93794c5fbfd2820e9acfedd9" } modify_external_catalogs: true
I20251029 08:52:24.009644 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4d8e69bc93794c5fbfd2820e9acfedd9" } modify_external_catalogs: true
I20251029 08:52:24.009111 1177 consensus_queue.cc:260] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.009996 861 leader_election.cc:304] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.010471 2451 raft_consensus.cc:2804] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.010586 2451 raft_consensus.cc:697] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:24.010787 2451 consensus_queue.cc:237] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.012118 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a4f861ca92fe4bb0af9cb605c5465744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.012616 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4f861ca92fe4bb0af9cb605c5465744 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.018740 2422 raft_consensus.cc:493] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.019001 2422 raft_consensus.cc:515] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.020288 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9429f6abb50c4f66b2e1b565f303962d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:24.020505 2422 leader_election.cc:290] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:24.020974 718 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.025043 1177 ts_tablet_manager.cc:1434] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
W20251029 08:52:24.025300 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 164415us
I20251029 08:52:24.026001 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 9493653c2e354458923e9e8dd0da0c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:24.026227 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:24.026415 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a92cc4a93a584dffaafe21316fe8dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:24.026384 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0ff0658e0274cf49c389ad602d91b03" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:24.026547 1069 raft_consensus.cc:3060] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.026602 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet fb22af7515084e099b41f9c1c92c067a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:24.027601 1177 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.029500 1069 raft_consensus.cc:2468] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:24.029994 1178 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:24.030233 1178 ts_tablet_manager.cc:1403] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.302s user 0.005s sys 0.000s
I20251029 08:52:24.031174 1178 raft_consensus.cc:359] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.035044 1974 ts_tablet_manager.cc:1929] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:24.048862 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b87d133f8a4bd1877e07a1a8758147" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.049523 1183 tablet_bootstrap.cc:492] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.052515 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9429f6abb50c4f66b2e1b565f303962d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.054759 2474 consensus_queue.cc:1048] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.057404 1183 ts_tablet_manager.cc:1403] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.193s user 0.004s sys 0.000s
I20251029 08:52:24.054836 2359 consensus_queue.cc:1048] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.058204 1183 raft_consensus.cc:359] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.058471 1178 raft_consensus.cc:385] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.058593 1178 raft_consensus.cc:740] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.058707 1183 raft_consensus.cc:385] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.058804 1183 raft_consensus.cc:740] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.059015 1183 consensus_queue.cc:260] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.059012 1178 consensus_queue.cc:260] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.059664 1183 ts_tablet_manager.cc:1434] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:24.060096 1183 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.061472 1178 ts_tablet_manager.cc:1434] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20251029 08:52:24.061937 1178 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:24.063824 1183 tablet_bootstrap.cc:654] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.066378 1178 tablet_bootstrap.cc:654] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.076229 1183 tablet_bootstrap.cc:492] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.076455 1183 ts_tablet_manager.cc:1403] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20251029 08:52:24.077239 1183 raft_consensus.cc:359] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.077559 1183 raft_consensus.cc:385] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.077673 1183 raft_consensus.cc:740] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.078013 1183 consensus_queue.cc:260] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.078604 1183 ts_tablet_manager.cc:1434] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251029 08:52:24.079015 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 33c7e63019634c229d824b2287683151 (table default.Table_af732cb4b90d49819386fe8dbb50ce73 [id=a0c554630ed843f894b51d8cc084cac4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 33c7e63019634c229d824b2287683151 has no consensus state
I20251029 08:52:24.079281 1183 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.081646 1183 tablet_bootstrap.cc:654] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.090221 1183 tablet_bootstrap.cc:492] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.090509 1183 ts_tablet_manager.cc:1403] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20251029 08:52:24.091362 1183 raft_consensus.cc:359] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.091620 1183 raft_consensus.cc:385] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.091732 1183 raft_consensus.cc:740] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.091969 1183 consensus_queue.cc:260] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.092772 1183 ts_tablet_manager.cc:1434] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:24.093171 1183 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.095330 1183 tablet_bootstrap.cc:654] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.100903 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d707b992dd69414897e525d86fb4b002" } modify_external_catalogs: true
I20251029 08:52:24.101131 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d707b992dd69414897e525d86fb4b002" } modify_external_catalogs: true
I20251029 08:52:24.102651 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79da38cb82164ec5acce7f9f37123031" } modify_external_catalogs: true
I20251029 08:52:24.102877 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_79da38cb82164ec5acce7f9f37123031" } modify_external_catalogs: true
I20251029 08:52:24.104305 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f117335b909e430dab2a5a5da90c9434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.105037 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f117335b909e430dab2a5a5da90c9434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.113588 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e687f593eead488db4bb445a63beeec7
I20251029 08:52:24.051574 1176 ts_tablet_manager.cc:1929] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:24.053406 720 leader_election.cc:343] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.114900 720 leader_election.cc:304] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.115209 2422 raft_consensus.cc:2749] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.115433 1176 log.cc:1199] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:24.115811 1176 ts_tablet_manager.cc:1950] T 3e65b3b1c8d14743aca1465d95ace95f P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:24.119087 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_17a9174cc94c45f1b499e06db7630b6f" } new_table_name: "default.Table_b66efc855f5b4b1ea6f2110c19c3812b" modify_external_catalogs: true
I20251029 08:52:24.121675 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 10baa0547180493f872a23614cf4c608
I20251029 08:52:24.127422 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet e687f593eead488db4bb445a63beeec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:24.127552 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8017e8fe5f634ef0bf7f8be7c8aea033 consensus state has no leader
I20251029 08:52:24.128047 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e15c46fe42ab4546913d8f009b074838"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.037545 1177 tablet_bootstrap.cc:654] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.079279 2451 consensus_queue.cc:1048] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.131623 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6722a3a5c0714d03b531866b14bfc7db" } new_table_name: "default.Table_98da77093ed14699a141e885b30e3088" modify_external_catalogs: true
I20251029 08:52:24.110877 2468 raft_consensus.cc:493] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.133643 2468 raft_consensus.cc:515] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.134270 2468 leader_election.cc:290] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.135036 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa411391c63146e7acb8b4c6edbe12a0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.135217 1065 raft_consensus.cc:2468] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:24.135962 861 leader_election.cc:304] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:24.136610 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c510c2d3e9184bcb92acb666b6bbfb84" } modify_external_catalogs: true
I20251029 08:52:24.136831 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c510c2d3e9184bcb92acb666b6bbfb84" } modify_external_catalogs: true
I20251029 08:52:24.137948 2468 raft_consensus.cc:2804] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.138031 2468 raft_consensus.cc:493] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.138093 2468 raft_consensus.cc:3060] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:24.139068 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2dbaf0f3d3242b388197299431cf86d has no consensus state
I20251029 08:52:24.141311 2468 raft_consensus.cc:515] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.142465 1177 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.142609 1177 ts_tablet_manager.cc:1403] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.000s
I20251029 08:52:24.143093 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa411391c63146e7acb8b4c6edbe12a0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:24.143301 1065 raft_consensus.cc:3060] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.143345 1177 raft_consensus.cc:359] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.143517 1177 raft_consensus.cc:385] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.143566 1177 raft_consensus.cc:740] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.143790 1177 consensus_queue.cc:260] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.144330 1177 ts_tablet_manager.cc:1434] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:24.144626 1177 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.146116 1177 tablet_bootstrap.cc:654] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.146329 1183 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.146579 1183 ts_tablet_manager.cc:1403] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.004s
I20251029 08:52:24.146883 1065 raft_consensus.cc:2468] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:24.147421 861 leader_election.cc:304] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:24.147743 2451 raft_consensus.cc:2804] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.147855 2451 raft_consensus.cc:697] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:24.148104 2451 consensus_queue.cc:237] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.147964 1183 raft_consensus.cc:359] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.148274 1183 raft_consensus.cc:385] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.148406 1183 raft_consensus.cc:740] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.148823 1183 consensus_queue.cc:260] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.150002 1183 ts_tablet_manager.cc:1434] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:24.150627 1183 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.151222 2359 raft_consensus.cc:493] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.151338 2359 raft_consensus.cc:515] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.151906 2359 leader_election.cc:290] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.152740 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.153339 996 leader_election.cc:343] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.154119 1183 tablet_bootstrap.cc:654] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.160192 2468 leader_election.cc:290] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.168210 1183 tablet_bootstrap.cc:492] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.168817 1183 ts_tablet_manager.cc:1403] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.002s
I20251029 08:52:24.170950 1974 log.cc:1199] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9e1c023b679b4c3298bf953b53d4eb96
I20251029 08:52:24.170851 1183 raft_consensus.cc:359] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.171250 1183 raft_consensus.cc:385] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.171403 1183 raft_consensus.cc:740] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.171631 1974 ts_tablet_manager.cc:1950] T 9e1c023b679b4c3298bf953b53d4eb96 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:24.172108 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 939) took 1734 ms. Trace:
I20251029 08:52:24.171989 1183 consensus_queue.cc:260] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.172235 1974 rpcz_store.cc:276] 1029 08:52:22.437154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.437701 (+ 547us) service_pool.cc:225] Handling call
1029 08:52:23.829175 (+1391474us) ts_tablet_manager.cc:1151] Deleting tablet 9e1c023b679b4c3298bf953b53d4eb96
1029 08:52:23.829189 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.835790 (+ 6601us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.034943 (+199153us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.172087 (+137144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391296}
I20251029 08:52:24.173082 1183 ts_tablet_manager.cc:1434] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 08:52:24.173162 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9e1c023b679b4c3298bf953b53d4eb96 (table default.Table_00d50b5cd5d045828ba7b366df301214 [id=b03188b6890542c392bd19534ace8bb0]) successfully deleted
I20251029 08:52:24.173631 1974 tablet_replica.cc:333] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:24.173767 1183 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.173945 1974 raft_consensus.cc:2243] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:24.174170 1974 raft_consensus.cc:2272] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:24.174865 1974 ts_tablet_manager.cc:1916] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:24.177861 1183 tablet_bootstrap.cc:654] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.180323 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 909) took 2178 ms. Trace:
I20251029 08:52:24.180498 1176 rpcz_store.cc:276] 1029 08:52:22.002049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.003472 (+ 1423us) service_pool.cc:225] Handling call
1029 08:52:23.686182 (+1682710us) ts_tablet_manager.cc:1151] Deleting tablet 3e65b3b1c8d14743aca1465d95ace95f
1029 08:52:23.686197 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:23.880761 (+194564us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.051448 (+170687us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.180266 (+128818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682547}
I20251029 08:52:24.181217 1176 tablet_replica.cc:333] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:24.181622 1176 raft_consensus.cc:2243] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:24.181963 1176 raft_consensus.cc:2272] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:24.183207 1176 ts_tablet_manager.cc:1916] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:24.189352 2359 raft_consensus.cc:493] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.189544 2359 raft_consensus.cc:515] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.190873 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.191530 996 leader_election.cc:343] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.192145 2359 leader_election.cc:290] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.192539 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66e4206742ae4022be3b1011f9bb49be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.198648 1183 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.199175 1183 ts_tablet_manager.cc:1403] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20251029 08:52:24.202744 2295 consensus_queue.cc:1048] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.114159 924 raft_consensus.cc:2468] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:24.115039 889 tablet_service.cc:1552] Processing DeleteTablet for tablet e687f593eead488db4bb445a63beeec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:24.203554 996 leader_election.cc:304] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.203819 2474 raft_consensus.cc:2804] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.203933 2474 raft_consensus.cc:697] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:24.204185 2474 consensus_queue.cc:237] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.202927 1183 raft_consensus.cc:359] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.205577 1183 raft_consensus.cc:385] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.205993 1183 raft_consensus.cc:740] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.207448 1183 consensus_queue.cc:260] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.211180 1183 ts_tablet_manager.cc:1434] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20251029 08:52:24.212700 1183 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:24.220177 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 166761us
I20251029 08:52:24.220858 754 tablet_service.cc:1552] Processing DeleteTablet for tablet e687f593eead488db4bb445a63beeec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:24.221192 720 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.221287 784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:24.221496 784 raft_consensus.cc:2468] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:24.221877 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:24.222023 777 raft_consensus.cc:2468] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:24.221297 720 leader_election.cc:304] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:24.224689 861 consensus_peers.cc:597] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41e435e9dfb34d4f8b0c8f7b702d5a6c. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:24.224954 861 leader_election.cc:343] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa411391c63146e7acb8b4c6edbe12a0
W20251029 08:52:24.225221 861 leader_election.cc:343] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:24.225729 2458 raft_consensus.cc:2749] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.228158 2156 raft_consensus.cc:493] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.228359 2156 raft_consensus.cc:515] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.220598 1183 tablet_bootstrap.cc:654] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.229111 2156 leader_election.cc:290] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.229593 2156 consensus_queue.cc:1048] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.222558 998 leader_election.cc:304] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:24.230547 998 consensus_peers.cc:597] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc04616301354cfe873c825a9f5d7ab6. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:24.230723 998 consensus_peers.cc:597] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aafa846cb2144cec9ce4b28e61d91961. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:24.230868 998 leader_election.cc:304] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.231321 2474 raft_consensus.cc:2804] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.231443 2474 raft_consensus.cc:493] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.231519 2474 raft_consensus.cc:3060] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.232700 2366 raft_consensus.cc:2804] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.232779 2366 raft_consensus.cc:493] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.232836 2366 raft_consensus.cc:3060] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.238657 2474 raft_consensus.cc:515] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.239329 2474 leader_election.cc:290] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.240646 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:24.240831 773 raft_consensus.cc:3060] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.241241 2366 raft_consensus.cc:515] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.241760 2366 leader_election.cc:290] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.242280 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:24.243127 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:24.243299 774 raft_consensus.cc:3060] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:24.243718 996 leader_election.cc:343] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.243829 773 raft_consensus.cc:2468] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:24.244400 998 leader_election.cc:304] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:24.244733 2366 raft_consensus.cc:2804] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.244846 2366 raft_consensus.cc:697] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:24.245115 2366 consensus_queue.cc:237] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.246452 774 raft_consensus.cc:2468] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:24.246842 998 leader_election.cc:304] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.251466 2474 raft_consensus.cc:2804] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.251766 2474 raft_consensus.cc:697] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:24.252547 2474 consensus_queue.cc:237] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.253240 1183 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.253654 1183 ts_tablet_manager.cc:1403] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.000s
I20251029 08:52:24.259056 1183 raft_consensus.cc:359] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.262063 1183 raft_consensus.cc:385] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.262634 1183 raft_consensus.cc:740] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.265398 1183 consensus_queue.cc:260] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.267419 2513 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:24.269209 1183 ts_tablet_manager.cc:1434] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s
I20251029 08:52:24.272073 1183 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.272229 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:24.273507 996 leader_election.cc:343] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:24.281292 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e15c46fe42ab4546913d8f009b074838 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.286329 1183 tablet_bootstrap.cc:654] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.287868 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c90eeb172173429785cab8c6674fbfcc" } new_table_name: "default.Table_9d4713e3019a4fedb6f8f00ed924536f" modify_external_catalogs: true
W20251029 08:52:24.296180 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 039ecc2142964a3782f6879b53f3b73e consensus state has no leader
I20251029 08:52:24.296595 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6e7b5c67ba3644c4be9ef7129ea50504" } new_table_name: "default.Table_af4260159d7346c49aabb31cfa9f224f" modify_external_catalogs: true
I20251029 08:52:24.301877 1183 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.302459 1183 ts_tablet_manager.cc:1403] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20251029 08:52:24.309120 1183 raft_consensus.cc:359] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.310016 1183 raft_consensus.cc:385] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:24.310534 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 900e8494364247289fa4208d81dfe3bb consensus state has no leader
I20251029 08:52:24.310493 1183 raft_consensus.cc:740] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.311625 1183 consensus_queue.cc:260] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.314003 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7e364318a48e4b8eb94a67fcd126a67b" } new_table_name: "default.Table_65ece294edec4bf395c6d737cfb14e29" modify_external_catalogs: true
I20251029 08:52:24.314695 1183 ts_tablet_manager.cc:1434] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251029 08:52:24.316442 1183 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.324762 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 10baa0547180493f872a23614cf4c608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:24.325352 1183 tablet_bootstrap.cc:654] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.325649 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 10baa0547180493f872a23614cf4c608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:24.326761 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_29c6685703864fd8bcb50adca11929d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.327397 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 10baa0547180493f872a23614cf4c608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:24.327759 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29c6685703864fd8bcb50adca11929d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:24.330610 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66e4206742ae4022be3b1011f9bb49be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.334558 1183 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.334720 1183 ts_tablet_manager.cc:1403] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251029 08:52:24.339494 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_982b0112dd78419e8ea54b9a114951b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.340159 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_982b0112dd78419e8ea54b9a114951b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.342221 1183 raft_consensus.cc:359] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.344480 1183 raft_consensus.cc:385] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.345592 1183 raft_consensus.cc:740] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.349038 1183 consensus_queue.cc:260] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.354049 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9310e1a601d94499a44877ea5a9dcc27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.354842 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9310e1a601d94499a44877ea5a9dcc27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.358253 1183 ts_tablet_manager.cc:1434] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.023s user 0.003s sys 0.002s
I20251029 08:52:24.362731 1183 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.363662 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_1d1ac2a940e44574bb336cd7805b6cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.364545 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d1ac2a940e44574bb336cd7805b6cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.366411 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_980d17e6af4a4439b1f459070584dcec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.367238 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_980d17e6af4a4439b1f459070584dcec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.372272 2468 raft_consensus.cc:493] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.372423 2468 raft_consensus.cc:515] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.373044 2468 leader_election.cc:290] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.385429 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6b04f7a83c444c96ab3363ca2d15709d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.386479 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b04f7a83c444c96ab3363ca2d15709d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.388860 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b9c2103e507b4beb9f03f4cbca60976d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.389506 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9c2103e507b4beb9f03f4cbca60976d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.390154 1183 tablet_bootstrap.cc:654] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.391149 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_683961d19f454703a28fc07f074d028d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.391955 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_683961d19f454703a28fc07f074d028d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.398702 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0d4cf693087a4d308c9e97704e60754d" } modify_external_catalogs: true
I20251029 08:52:24.398981 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0d4cf693087a4d308c9e97704e60754d" } modify_external_catalogs: true
I20251029 08:52:24.428699 2485 raft_consensus.cc:493] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.428900 2485 raft_consensus.cc:515] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.429697 2485 leader_election.cc:290] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.430231 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:24.430460 774 raft_consensus.cc:2393] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 730d0f5b7abf43a293b7df13d01ce4ee in current term 1: Already voted for candidate 927689e0dc5e4bc79d4a9d79a873f088 in this term.
I20251029 08:52:24.432154 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "930314c8f90f462990f93fab8f7790c2" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:24.435536 996 leader_election.cc:304] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.436105 2485 raft_consensus.cc:3060] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.442533 2485 raft_consensus.cc:2749] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.448331 1183 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.448529 1183 ts_tablet_manager.cc:1403] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.004s
I20251029 08:52:24.456270 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c8619310a8224be58710ffee5aa94c98" } new_table_name: "default.Table_413b087c2c7b449096ad12931c933d83" modify_external_catalogs: true
I20251029 08:52:24.456408 1183 raft_consensus.cc:359] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.458907 1183 raft_consensus.cc:385] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.459770 1183 raft_consensus.cc:740] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.461190 1183 consensus_queue.cc:260] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.461860 1183 ts_tablet_manager.cc:1434] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20251029 08:52:24.462234 1183 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:24.471987 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 246529us
I20251029 08:52:24.472664 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd0c43367a8c47f390cc61d2764d61e4" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.472844 1065 raft_consensus.cc:2468] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:24.474390 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "652903cbbdb042d780e7d71fc2c0ab4d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.474525 1065 raft_consensus.cc:2468] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:24.476380 2468 raft_consensus.cc:493] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.476513 2468 raft_consensus.cc:515] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.477070 2468 leader_election.cc:290] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.478260 2513 tablet.cc:1722] T 2d11b75de7d448c68fd63ab1ae81a464 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:24.490285 2523 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:24.492503 1178 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:24.492650 1178 ts_tablet_manager.cc:1403] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.431s user 0.003s sys 0.001s
I20251029 08:52:24.493093 861 leader_election.cc:304] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:24.493494 2295 raft_consensus.cc:2804] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.493569 2295 raft_consensus.cc:493] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.493639 2295 raft_consensus.cc:3060] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.493570 1178 raft_consensus.cc:359] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.493733 1178 raft_consensus.cc:385] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.493780 1178 raft_consensus.cc:740] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.493968 1178 consensus_queue.cc:260] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.494417 1178 ts_tablet_manager.cc:1434] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:24.494663 1178 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:24.495057 2476 raft_consensus.cc:493] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.495232 2476 raft_consensus.cc:515] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.496080 2476 leader_election.cc:290] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.496361 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa0ff14e19b4ad7a8b33796481f0f3f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.496767 996 leader_election.cc:343] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:24.497150 998 leader_election.cc:343] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4aa0ff14e19b4ad7a8b33796481f0f3f
I20251029 08:52:24.497278 998 leader_election.cc:304] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.497732 2476 raft_consensus.cc:2749] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.502470 2295 raft_consensus.cc:515] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.503028 2295 leader_election.cc:290] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.506155 861 leader_election.cc:304] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:24.507155 2295 raft_consensus.cc:2804] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.507277 2295 raft_consensus.cc:493] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.507385 2295 raft_consensus.cc:3060] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.510339 1183 tablet_bootstrap.cc:654] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.511070 2295 raft_consensus.cc:515] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.511649 2476 consensus_queue.cc:1048] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.512697 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6bfec7dc813f42d588da6bd24fe2f3b7" } new_table_name: "default.Table_a83c7d283315417781f403f77ef76951" modify_external_catalogs: true
I20251029 08:52:24.520970 2295 leader_election.cc:290] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.524570 2468 consensus_queue.cc:1048] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:24.524938 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 50095.5us
W20251029 08:52:24.528650 861 leader_election.cc:343] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 652903cbbdb042d780e7d71fc2c0ab4d
W20251029 08:52:24.530891 861 leader_election.cc:343] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bd0c43367a8c47f390cc61d2764d61e4
I20251029 08:52:24.531644 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43c3074caee74069a8a7f1b87721d726" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.538741 2531 tablet.cc:1722] T 2d11b75de7d448c68fd63ab1ae81a464 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:24.541992 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d9face4d3c64b8987085cd5b8662db9 consensus state has no leader
I20251029 08:52:24.542785 591 master_service.cc:954] ReplaceTablet: received request to replace tablet a73a891fb6034f989d29a3f837303d84 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:24.553331 1177 tablet_bootstrap.cc:492] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.553650 1177 ts_tablet_manager.cc:1403] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.409s user 0.104s sys 0.153s
I20251029 08:52:24.554529 1177 raft_consensus.cc:359] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.554823 1177 raft_consensus.cc:385] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.554924 1177 raft_consensus.cc:740] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.555193 1177 consensus_queue.cc:260] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.555733 1177 ts_tablet_manager.cc:1434] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:24.556083 1177 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.557782 1177 tablet_bootstrap.cc:654] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.558151 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd0c43367a8c47f390cc61d2764d61e4" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:24.558329 1067 raft_consensus.cc:3060] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:24.560439 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 211f3fde17f64c99a0b80e377c028fb4 consensus state has no leader
I20251029 08:52:24.561017 1067 raft_consensus.cc:2468] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:24.561506 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2b1a82f91b4542a7b759c6efd549e4ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.562243 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b1a82f91b4542a7b759c6efd549e4ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.563588 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "652903cbbdb042d780e7d71fc2c0ab4d" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:24.564385 1065 raft_consensus.cc:3060] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.567771 1065 raft_consensus.cc:2468] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:24.576725 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a73a891fb6034f989d29a3f837303d84
I20251029 08:52:24.576874 1177 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.577029 1177 ts_tablet_manager.cc:1403] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 08:52:24.577658 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet a73a891fb6034f989d29a3f837303d84 with delete_type TABLET_DATA_DELETED (replaced by tablet a711e95fbef84b12997d01c1883057d9) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:24.577739 1177 raft_consensus.cc:359] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.577921 1177 raft_consensus.cc:385] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.577977 1177 raft_consensus.cc:740] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.578192 1177 consensus_queue.cc:260] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.578289 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a73a891fb6034f989d29a3f837303d84 with delete_type TABLET_DATA_DELETED (replaced by tablet a711e95fbef84b12997d01c1883057d9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:24.578660 1177 ts_tablet_manager.cc:1434] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:24.578943 1177 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.580626 1177 tablet_bootstrap.cc:654] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.582098 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:24.588512 889 tablet_service.cc:1552] Processing DeleteTablet for tablet a73a891fb6034f989d29a3f837303d84 with delete_type TABLET_DATA_DELETED (replaced by tablet a711e95fbef84b12997d01c1883057d9) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:24.601202 861 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726
W20251029 08:52:24.602191 861 consensus_peers.cc:597] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2b7853b85d24a04b8b11caaa4fe8c4d. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:24.603232 861 leader_election.cc:343] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bd0c43367a8c47f390cc61d2764d61e4
W20251029 08:52:24.604507 861 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.604710 861 leader_election.cc:304] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.605022 1178 tablet_bootstrap.cc:654] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.605034 2517 raft_consensus.cc:2749] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.609488 1177 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.609702 1177 ts_tablet_manager.cc:1403] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20251029 08:52:24.611330 1183 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.611843 1183 ts_tablet_manager.cc:1403] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.014s
I20251029 08:52:24.616328 1183 raft_consensus.cc:359] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.616964 1183 raft_consensus.cc:385] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.617213 1183 raft_consensus.cc:740] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.617974 1183 consensus_queue.cc:260] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.619997 1183 ts_tablet_manager.cc:1434] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20251029 08:52:24.620273 998 consensus_peers.cc:597] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af38662420d14ac1a7cb5fd03fc1b9d8. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:24.620975 1183 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.622941 1176 ts_tablet_manager.cc:1929] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:24.623070 1176 log.cc:1199] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:24.623364 1176 ts_tablet_manager.cc:1950] T c9bbf6401423411089c9070f3b03f1a7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:24.626603 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 912) took 2607 ms. Trace:
I20251029 08:52:24.626701 1176 rpcz_store.cc:276] 1029 08:52:22.019143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.019958 (+ 815us) service_pool.cc:225] Handling call
1029 08:52:24.181098 (+2161140us) ts_tablet_manager.cc:1151] Deleting tablet c9bbf6401423411089c9070f3b03f1a7
1029 08:52:24.181140 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.550045 (+368905us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.622818 (+ 72773us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.626583 (+ 3765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2160920}
I20251029 08:52:24.626842 1183 tablet_bootstrap.cc:654] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.627192 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c9bbf6401423411089c9070f3b03f1a7 (table default.Table_fef23ec2ce374eadbd727deb26c45850 [id=58aabf0c50d64caa918b4d97574d7c5f]) successfully deleted
I20251029 08:52:24.627594 591 catalog_manager.cc:6719] ReplaceTablet: tablet a73a891fb6034f989d29a3f837303d84 deleted and replaced by tablet a711e95fbef84b12997d01c1883057d9
I20251029 08:52:24.628393 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ffc61d294dca458aba887c7e9614a720" } new_table_name: "default.Table_06720bad084b461996fdf83b38285cd3" modify_external_catalogs: true
I20251029 08:52:24.633081 1177 raft_consensus.cc:359] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.633262 1177 raft_consensus.cc:385] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.633313 1177 raft_consensus.cc:740] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.633579 1177 consensus_queue.cc:260] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.634030 1177 ts_tablet_manager.cc:1434] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251029 08:52:24.635219 1177 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.637005 1177 tablet_bootstrap.cc:654] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.651392 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ae84d579cd964855a5e44aa279312302" } new_table_name: "default.Table_5329de213c3e4ae088e1a8d1674186db" modify_external_catalogs: true
I20251029 08:52:24.651731 861 leader_election.cc:304] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.652374 2295 raft_consensus.cc:2804] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.652500 2295 raft_consensus.cc:697] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:24.652747 2295 consensus_queue.cc:237] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.658572 1974 ts_tablet_manager.cc:1929] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:24.658771 1974 log.cc:1199] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/75e1f0cdfbc34c6f85f2ad545eb33c0d
I20251029 08:52:24.659159 1974 ts_tablet_manager.cc:1950] T 75e1f0cdfbc34c6f85f2ad545eb33c0d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:24.659561 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 941) took 2222 ms. Trace:
I20251029 08:52:24.659709 1974 rpcz_store.cc:276] 1029 08:52:22.437363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.439576 (+ 2213us) service_pool.cc:225] Handling call
1029 08:52:24.173568 (+1733992us) ts_tablet_manager.cc:1151] Deleting tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d
1029 08:52:24.173579 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.558867 (+385288us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.658329 (+ 99462us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.659541 (+ 1212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733828}
I20251029 08:52:24.660635 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 75e1f0cdfbc34c6f85f2ad545eb33c0d (table default.Table_a2841227adaf4c67aed78c15574f5188 [id=8dff11bc317244c693f3b7113f96ed32]) successfully deleted
I20251029 08:52:24.661226 861 leader_election.cc:304] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:24.662328 2458 raft_consensus.cc:493] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.662521 2458 raft_consensus.cc:515] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.663156 2458 leader_election.cc:290] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:24.664780 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb380018f46465e82ac6223ecf8dd0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.665318 718 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.665827 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb380018f46465e82ac6223ecf8dd0d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:24.666823 720 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.666929 720 leader_election.cc:304] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.667371 2458 raft_consensus.cc:2749] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.668119 2517 raft_consensus.cc:2804] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:24.668393 2517 raft_consensus.cc:697] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:24.671859 1183 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.672542 1183 ts_tablet_manager.cc:1403] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.003s
I20251029 08:52:24.674500 2468 raft_consensus.cc:493] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.674636 2468 raft_consensus.cc:515] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.674533 1183 raft_consensus.cc:359] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.674957 1183 raft_consensus.cc:385] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.675170 2468 leader_election.cc:290] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.675109 1183 raft_consensus.cc:740] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:24.676208 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b7a6008b4e6243ccaf166a31f1a9f62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6008b4e6243ccaf166a31f1a9f62c already in progress: creating tablet
I20251029 08:52:24.678961 2366 raft_consensus.cc:493] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.677619 1183 consensus_queue.cc:260] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.681346 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f5ebd2bf6614980ac0c18f5d9974fc0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.681140 1178 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:24.682471 1178 ts_tablet_manager.cc:1403] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.188s user 0.001s sys 0.004s
W20251029 08:52:24.684000 861 leader_election.cc:343] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 652903cbbdb042d780e7d71fc2c0ab4d
W20251029 08:52:24.684358 861 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
W20251029 08:52:24.684563 861 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.684686 861 leader_election.cc:304] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.684587 1183 ts_tablet_manager.cc:1434] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.012s user 0.003s sys 0.001s
I20251029 08:52:24.679196 2366 raft_consensus.cc:515] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:24.688905 998 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:24.688548 1183 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.691920 2156 raft_consensus.cc:2749] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.693425 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa4f97075b3941aaa7cda3f9413e867f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.694015 996 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.694111 996 leader_election.cc:304] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.694290 2476 raft_consensus.cc:2749] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.694944 2366 leader_election.cc:290] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:24.693310 1178 raft_consensus.cc:359] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.696070 1178 raft_consensus.cc:385] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.697080 1178 raft_consensus.cc:740] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.700078 1178 consensus_queue.cc:260] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:24.703404 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b61829fcd59d4ad4ab427270b7089d57 consensus state has no leader
I20251029 08:52:24.706513 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c9c0266a3d25476cae75216163270a5f" } modify_external_catalogs: true
I20251029 08:52:24.711483 2545 tablet.cc:1722] T 2d11b75de7d448c68fd63ab1ae81a464 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:24.710757 2536 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:24.712337 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c9c0266a3d25476cae75216163270a5f" } modify_external_catalogs: true
I20251029 08:52:24.713793 1178 ts_tablet_manager.cc:1434] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.030s user 0.001s sys 0.003s
W20251029 08:52:24.715780 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef67d5cbc902492fb1252229b4b7f28f consensus state has no leader
I20251029 08:52:24.715821 1183 tablet_bootstrap.cc:654] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.715992 1178 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:24.669296 2517 consensus_queue.cc:237] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.718494 2513 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:24.721695 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc4c12b3775849c8a71e324ebbc50098" } new_table_name: "default.Table_bceccce7081b43f6ad2849ed269dfe73" modify_external_catalogs: true
I20251029 08:52:24.725351 2523 tablet.cc:1722] T 898a4f58306b453e9abca2a73082bb96 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:24.726953 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 943) took 2230 ms. Trace:
I20251029 08:52:24.727165 1974 rpcz_store.cc:276] 1029 08:52:22.496668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.496738 (+ 70us) service_pool.cc:225] Handling call
1029 08:52:24.714502 (+2217764us) ts_tablet_manager.cc:1151] Deleting tablet 42de32d87ebb451090e0310e36e64277
1029 08:52:24.714516 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.726904 (+ 12388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2217230}
W20251029 08:52:24.727790 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 42de32d87ebb451090e0310e36e64277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42de32d87ebb451090e0310e36e64277 already in progress: creating tablet
I20251029 08:52:24.728042 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 944) took 2223 ms. Trace:
I20251029 08:52:24.728354 1974 rpcz_store.cc:276] 1029 08:52:22.504170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.504369 (+ 199us) service_pool.cc:225] Handling call
1029 08:52:24.727835 (+2223466us) ts_tablet_manager.cc:1151] Deleting tablet 28d62d1e354d41f49b9ae30dbb6efc0f
1029 08:52:24.727847 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.727992 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2223231}
W20251029 08:52:24.728873 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 28d62d1e354d41f49b9ae30dbb6efc0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28d62d1e354d41f49b9ae30dbb6efc0f already in progress: creating tablet
I20251029 08:52:24.729009 1974 tablet_replica.cc:333] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:24.729281 1974 raft_consensus.cc:2243] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:24.729702 1974 pending_rounds.cc:70] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:24.729835 1974 pending_rounds.cc:77] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037657756418048 op_type: NO_OP noop_request { }
I20251029 08:52:24.730739 1974 raft_consensus.cc:2889] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:24.730904 1974 raft_consensus.cc:2272] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:24.731406 2458 consensus_queue.cc:1048] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.732455 1974 ts_tablet_manager.cc:1916] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:24.735085 1178 tablet_bootstrap.cc:654] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.737031 1176 tablet_replica.cc:333] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:24.737282 1176 raft_consensus.cc:2243] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:24.737491 1176 raft_consensus.cc:2272] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:24.738150 1176 ts_tablet_manager.cc:1916] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:24.739437 2513 tablet.cc:1722] T 898a4f58306b453e9abca2a73082bb96 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:24.749868 1179 tablet_replica.cc:333] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:24.750239 1179 raft_consensus.cc:2243] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:24.750532 1179 raft_consensus.cc:2272] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:24.751201 1179 ts_tablet_manager.cc:1916] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:24.762480 2476 consensus_queue.cc:1048] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.762959 2476 raft_consensus.cc:493] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.763090 2476 raft_consensus.cc:515] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.763665 2476 leader_election.cc:290] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.764993 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39fc9612689b46d498d6092bd552da4e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.765543 996 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:24.766026 998 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 39fc9612689b46d498d6092bd552da4e
I20251029 08:52:24.766139 998 leader_election.cc:304] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.766726 2476 raft_consensus.cc:2749] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.767635 1178 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:24.769431 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
W20251029 08:52:24.774333 998 consensus_peers.cc:597] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed66c3fa3f848df905a2a31fd553474. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:24.783646 2366 raft_consensus.cc:493] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.783677 1177 tablet_bootstrap.cc:492] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:24.783854 1177 ts_tablet_manager.cc:1403] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.024s
I20251029 08:52:24.783857 2366 raft_consensus.cc:515] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:24.785789 998 leader_election.cc:343] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:24.785430 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c4d793f10147fba5e4f44474575bf0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.786449 996 leader_election.cc:343] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.786557 996 leader_election.cc:304] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.786950 2476 raft_consensus.cc:2749] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.787124 2366 leader_election.cc:290] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:24.790861 1177 raft_consensus.cc:359] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.796319 1177 raft_consensus.cc:385] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.796455 1177 raft_consensus.cc:740] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.796773 1177 consensus_queue.cc:260] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:24.807587 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c2072c16fce43278f1e32b9672187e7 consensus state has no leader
I20251029 08:52:24.808497 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_073a0e8a44f04947b524dac0cdb01fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.809623 1177 ts_tablet_manager.cc:1434] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.024s user 0.001s sys 0.003s
I20251029 08:52:24.809917 1177 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:24.811707 1177 tablet_bootstrap.cc:654] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.811949 1178 ts_tablet_manager.cc:1403] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.004s
I20251029 08:52:24.812814 1178 raft_consensus.cc:359] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.813071 1178 raft_consensus.cc:385] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.813211 1178 raft_consensus.cc:740] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.813975 1178 consensus_queue.cc:260] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.815598 1178 ts_tablet_manager.cc:1434] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:24.822240 1178 tablet_bootstrap.cc:492] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:24.828747 1183 tablet_bootstrap.cc:492] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.829016 1183 ts_tablet_manager.cc:1403] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.142s user 0.002s sys 0.011s
I20251029 08:52:24.829732 1183 raft_consensus.cc:359] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.829902 1183 raft_consensus.cc:385] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.829950 1183 raft_consensus.cc:740] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.830142 1183 consensus_queue.cc:260] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.830732 1183 ts_tablet_manager.cc:1434] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:24.830989 1183 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.832603 1183 tablet_bootstrap.cc:654] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:24.841532 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073a0e8a44f04947b524dac0cdb01fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.849483 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b4b6fc6f5ffa4c95b59c20c1aa578af9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:24.850119 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4b6fc6f5ffa4c95b59c20c1aa578af9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.850461 1178 tablet_bootstrap.cc:654] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:24.854945 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 58095.3us
I20251029 08:52:24.857291 2458 raft_consensus.cc:493] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.857441 2458 raft_consensus.cc:515] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.858103 2458 leader_election.cc:290] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:24.859026 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0a69ab65d26044739b602d6040ee95a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.859851 2523 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:24.860781 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9adbcf3f12104fb5bfd1d9dd5c946c54" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:24.860932 861 consensus_peers.cc:597] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:24.861085 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a69ab65d26044739b602d6040ee95a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:24.861279 861 consensus_peers.cc:597] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:24.861665 720 leader_election.cc:343] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.862221 2545 tablet.cc:1722] T 898a4f58306b453e9abca2a73082bb96 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:24.866672 2545 tablet.cc:1722] T cc57cd5a8d3c42ba90f2011084e8b4bd P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:24.880391 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9adbcf3f12104fb5bfd1d9dd5c946c54" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.880435 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 88421.2us
I20251029 08:52:24.882581 1183 tablet_bootstrap.cc:492] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.882754 1183 ts_tablet_manager.cc:1403] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.007s
I20251029 08:52:24.883340 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_39789940cb7e4119892e076e8929054f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.883422 1183 raft_consensus.cc:359] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.883612 1183 raft_consensus.cc:385] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.883674 1183 raft_consensus.cc:740] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.883896 1183 consensus_queue.cc:260] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:24.884065 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39789940cb7e4119892e076e8929054f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.884567 1183 ts_tablet_manager.cc:1434] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:24.884874 1183 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.886722 1183 tablet_bootstrap.cc:654] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:24.899906 1183 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:24.900099 1183 ts_tablet_manager.cc:1403] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20251029 08:52:24.900935 1183 raft_consensus.cc:359] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.901113 1183 raft_consensus.cc:385] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.901166 1183 raft_consensus.cc:740] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.901353 1183 consensus_queue.cc:260] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.904377 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_39fcfbbc91a84ece9939c873e4a33e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.906107 1183 ts_tablet_manager.cc:1434] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251029 08:52:24.906308 2555 raft_consensus.cc:493] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.906436 1183 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:24.906443 2555 raft_consensus.cc:515] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.908807 1183 tablet_bootstrap.cc:654] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:24.909516 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39fcfbbc91a84ece9939c873e4a33e9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:24.884742 718 leader_election.cc:343] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.913581 718 leader_election.cc:304] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.914994 2422 raft_consensus.cc:2749] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.883783 1176 ts_tablet_manager.cc:1929] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:24.918828 1176 log.cc:1199] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:24.919510 1176 ts_tablet_manager.cc:1950] T 254e62d88c3242a0a3ac7470a8005d74 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:24.921495 2458 consensus_queue.cc:1048] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:24.927142 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2696186101b6433885b1929577db93d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:24.929054 1178 tablet_bootstrap.cc:492] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:24.929617 1178 ts_tablet_manager.cc:1403] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.002s
I20251029 08:52:24.930533 1178 raft_consensus.cc:359] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.930799 1178 raft_consensus.cc:385] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:24.930915 1178 raft_consensus.cc:740] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:24.931192 1178 consensus_queue.cc:260] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:24.934435 2359 raft_consensus.cc:493] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.934623 2359 raft_consensus.cc:515] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.935010 1178 ts_tablet_manager.cc:1434] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:24.935418 2359 leader_election.cc:290] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:24.935866 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2696186101b6433885b1929577db93d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:24.936206 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba201a54466240129d82a096f9316242" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.936895 996 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:24.937028 998 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ba201a54466240129d82a096f9316242
I20251029 08:52:24.937171 998 leader_election.cc:304] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.937606 2475 raft_consensus.cc:2749] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.944691 2555 leader_election.cc:290] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.948531 2554 raft_consensus.cc:493] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.948675 2554 raft_consensus.cc:515] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:24.951004 861 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c4d57a59684723bfd27c7dda302ef3
W20251029 08:52:24.951601 998 consensus_peers.cc:597] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbc4345cabbc476485b6a040a076ddaf. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:24.952072 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c4d57a59684723bfd27c7dda302ef3" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:24.952765 861 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.952893 861 leader_election.cc:304] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.953248 2555 raft_consensus.cc:2749] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.954694 2554 leader_election.cc:290] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.956399 1179 ts_tablet_manager.cc:1929] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:24.956532 1179 log.cc:1199] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:24.956751 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:24.956892 1179 ts_tablet_manager.cc:1950] T de78ea608c904aeea7f7145b9c80f45b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:24.956988 778 raft_consensus.cc:2393] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:24.957226 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 897) took 2802 ms. Trace:
I20251029 08:52:24.957331 1179 rpcz_store.cc:276] 1029 08:52:22.155107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.155726 (+ 619us) service_pool.cc:225] Handling call
1029 08:52:24.749795 (+2594069us) ts_tablet_manager.cc:1151] Deleting tablet de78ea608c904aeea7f7145b9c80f45b
1029 08:52:24.749808 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.817986 (+ 68178us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.956258 (+138272us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.957209 (+ 951us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593917}
I20251029 08:52:24.957821 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713e102d9e1d40148636443a2260ccbc" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:24.957862 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet de78ea608c904aeea7f7145b9c80f45b (table default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]) successfully deleted
I20251029 08:52:24.958351 861 leader_election.cc:304] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:24.958727 2554 raft_consensus.cc:3060] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.959642 2475 raft_consensus.cc:493] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.959767 2475 raft_consensus.cc:515] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.960330 2475 leader_election.cc:290] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.960755 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.961349 996 leader_election.cc:343] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.962073 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:24.962339 778 raft_consensus.cc:2468] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:24.962822 998 leader_election.cc:304] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:24.963258 2475 raft_consensus.cc:2804] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:24.963335 2475 raft_consensus.cc:493] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:24.963390 2475 raft_consensus.cc:3060] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:24.967798 2475 raft_consensus.cc:515] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.969079 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:24.970052 996 leader_election.cc:343] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.970872 2475 leader_election.cc:290] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:24.974757 2554 raft_consensus.cc:2749] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:24.988127 2475 raft_consensus.cc:493] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:24.988369 2475 raft_consensus.cc:515] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:24.989475 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:24.990047 996 leader_election.cc:343] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:24.990797 2475 leader_election.cc:290] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.004617 1177 tablet_bootstrap.cc:492] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.004854 1177 ts_tablet_manager.cc:1403] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.195s user 0.001s sys 0.044s
I20251029 08:52:25.007442 2554 consensus_queue.cc:1048] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.012130 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 25bd82e3549442ada2503518037127fc
I20251029 08:52:25.012600 589 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39506 (request call id 1185) took 1879 ms. Trace:
I20251029 08:52:25.012697 589 rpcz_store.cc:276] 1029 08:52:23.132925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.683802 (+550877us) service_pool.cc:225] Handling call
1029 08:52:23.690891 (+ 7089us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb914ff9256 00007fb90bb0a486 00007fb90bb0a1b5 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:23.691246 (+ 355us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1029 08:52:23.691374 (+ 128us) catalog_manager.cc:2815] Modifying in-memory table state
1029 08:52:23.691439 (+ 65us) catalog_manager.cc:2824] Locking tablets
1029 08:52:25.005058 (+1313619us) catalog_manager.cc:2838] Removing table and tablets from system table
1029 08:52:25.011951 (+ 6893us) catalog_manager.cc:2856] Removing table from by-name map
1029 08:52:25.012053 (+ 102us) catalog_manager.cc:2872] Committing in-memory state
1029 08:52:25.012088 (+ 35us) catalog_manager.cc:2876] Aborting table tasks
1029 08:52:25.012549 (+ 461us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.005814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.005919 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.005924 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:25.005926 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:25.006021 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.006031 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.006035 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:25.006125 (+ 90us) tablet.cc:699] Row locks acquired
1029 08:52:25.006127 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:25.006160 (+ 33us) write_op.cc:270] Start()
1029 08:52:25.006187 (+ 27us) write_op.cc:276] Timestamp: P: 1761727945006122 usec, L: 0
1029 08:52:25.006192 (+ 5us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.006358 (+ 166us) log.cc:844] Serialized 725 byte log entry
1029 08:52:25.010945 (+ 4587us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.011366 (+ 421us) write_op.cc:301] APPLY: starting
1029 08:52:25.011379 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.011433 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.011455 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.011507 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.011539 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 08:52:25.011546 (+ 7us) write_op.cc:312] APPLY: finished
1029 08:52:25.011619 (+ 73us) log.cc:844] Serialized 29 byte log entry
1029 08:52:25.011671 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.011685 (+ 14us) write_op.cc:454] Released schema lock
1029 08:52:25.011690 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:25.011751 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.011755 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":15808768,"child_traces":[["op",{"apply.queue_time_us":367,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":380,"num_ops":2,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":676,"prepare.run_wall_time_us":677,"raft.queue_time_us":62,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":4722}]]}
I20251029 08:52:25.005668 1177 raft_consensus.cc:359] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.016383 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 25bd82e3549442ada2503518037127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.016649 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 25bd82e3549442ada2503518037127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.016079 1177 raft_consensus.cc:385] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.017702 1183 tablet_bootstrap.cc:492] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.017891 1183 ts_tablet_manager.cc:1403] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.112s user 0.016s sys 0.008s
I20251029 08:52:25.017210 1177 raft_consensus.cc:740] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.018447 1177 consensus_queue.cc:260] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.018908 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 25bd82e3549442ada2503518037127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:23 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.018952 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:25.018954 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:25.019162 778 raft_consensus.cc:3060] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.019179 773 raft_consensus.cc:2468] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:25.019467 861 consensus_peers.cc:597] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de78ea608c904aeea7f7145b9c80f45b. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.019780 998 leader_election.cc:304] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.020898 2475 raft_consensus.cc:2804] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.021235 2475 raft_consensus.cc:493] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.021914 778 raft_consensus.cc:2468] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.021814 2475 raft_consensus.cc:3060] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.022804 998 leader_election.cc:304] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.023048 2359 raft_consensus.cc:2804] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.023149 2359 raft_consensus.cc:697] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.023375 2359 consensus_queue.cc:237] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.024160 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5f67e7b28b4240618e1817bb47d5633d
W20251029 08:52:25.024555 998 consensus_peers.cc:597] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b5051ff776e4975b917db0393a1f945. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.024637 593 catalog_manager.cc:6719] ReplaceTablet: tablet 5f67e7b28b4240618e1817bb47d5633d deleted and replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04
I20251029 08:52:25.024752 593 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39506 (request call id 1205) took 1049 ms. Trace:
I20251029 08:52:25.024858 593 rpcz_store.cc:276] 1029 08:52:23.974807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.002395 (+ 27588us) service_pool.cc:225] Handling call
1029 08:52:24.011797 (+ 9402us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb915004fab 00007fb90bb15107 00007fb90bb14e35 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:25.024717 (+1012920us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.007146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.007251 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.007256 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:25.007258 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:25.007351 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.007360 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.007364 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:25.007450 (+ 86us) tablet.cc:699] Row locks acquired
1029 08:52:25.007452 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:25.007483 (+ 31us) write_op.cc:270] Start()
1029 08:52:25.007507 (+ 24us) write_op.cc:276] Timestamp: P: 1761727945007445 usec, L: 0
1029 08:52:25.007510 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.007663 (+ 153us) log.cc:844] Serialized 622 byte log entry
1029 08:52:25.019908 (+ 12245us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.023000 (+ 3092us) write_op.cc:301] APPLY: starting
1029 08:52:25.023019 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.023062 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.023101 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.023207 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.023264 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:25.023271 (+ 7us) write_op.cc:312] APPLY: finished
1029 08:52:25.023352 (+ 81us) log.cc:844] Serialized 29 byte log entry
1029 08:52:25.023441 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.023461 (+ 20us) write_op.cc:454] Released schema lock
1029 08:52:25.023468 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:25.023538 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.023541 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":20757760,"child_traces":[["op",{"apply.queue_time_us":3031,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":534,"num_ops":2,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":578,"replication_time_us":12371}]]}
I20251029 08:52:25.026583 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f67e7b28b4240618e1817bb47d5633d with delete_type TABLET_DATA_DELETED (replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.026971 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 667c71a8dfd54c05ab56e845653c0c9a
I20251029 08:52:25.028648 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.032377 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.033005 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f67e7b28b4240618e1817bb47d5633d with delete_type TABLET_DATA_DELETED (replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.034755 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f67e7b28b4240618e1817bb47d5633d with delete_type TABLET_DATA_DELETED (replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.035136 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 667c71a8dfd54c05ab56e845653c0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.035147 1974 ts_tablet_manager.cc:1929] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.035264 1974 log.cc:1199] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/930314c8f90f462990f93fab8f7790c2
I20251029 08:52:25.035521 2485 consensus_queue.cc:1048] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.035778 1974 ts_tablet_manager.cc:1950] T 930314c8f90f462990f93fab8f7790c2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:25.035799 998 consensus_peers.cc:597] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24ab9df6f768410eb4f66cffdcd68df2. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.036141 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 949) took 2482 ms. Trace:
I20251029 08:52:25.036224 1974 rpcz_store.cc:276] 1029 08:52:22.554161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.554234 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:24.728907 (+2174673us) ts_tablet_manager.cc:1151] Deleting tablet 930314c8f90f462990f93fab8f7790c2
1029 08:52:24.728943 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.779299 (+ 50356us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.035024 (+255725us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.036124 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174463}
I20251029 08:52:25.036742 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 930314c8f90f462990f93fab8f7790c2 (table default.Table_11e47ff2a71e439ebfb0e9ba79a95267 [id=5691028224ad4d21b0c0ea667274b8af]) successfully deleted
I20251029 08:52:25.037211 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 950) took 2472 ms. Trace:
I20251029 08:52:25.037261 1178 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.038416 1183 raft_consensus.cc:359] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.038699 1183 raft_consensus.cc:385] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.038043 2475 raft_consensus.cc:515] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.039207 1178 tablet_bootstrap.cc:654] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.038836 1183 raft_consensus.cc:740] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.040728 1183 consensus_queue.cc:260] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.041306 1183 ts_tablet_manager.cc:1434] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251029 08:52:25.041616 1183 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.043687 1178 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.043818 1178 ts_tablet_manager.cc:1403] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20251029 08:52:25.044211 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:25.044359 783 raft_consensus.cc:3060] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.044468 2475 leader_election.cc:290] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.044657 1178 raft_consensus.cc:359] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.044790 1178 raft_consensus.cc:385] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.044828 1178 raft_consensus.cc:740] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.045010 1178 consensus_queue.cc:260] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.045439 1178 ts_tablet_manager.cc:1434] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:25.045676 1178 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.047194 783 raft_consensus.cc:2468] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.047262 1178 tablet_bootstrap.cc:654] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.043998 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 913) took 3024 ms. Trace:
I20251029 08:52:25.047734 1176 rpcz_store.cc:276] 1029 08:52:22.019212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.020206 (+ 994us) service_pool.cc:225] Handling call
1029 08:52:24.736978 (+2716772us) ts_tablet_manager.cc:1151] Deleting tablet 254e62d88c3242a0a3ac7470a8005d74
1029 08:52:24.736990 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:24.813740 (+ 76750us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:24.883651 (+ 69911us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.043982 (+160331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2716653}
I20251029 08:52:25.048316 998 leader_election.cc:304] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:25.048617 2475 raft_consensus.cc:2804] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.048758 2475 raft_consensus.cc:697] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.049032 2475 consensus_queue.cc:237] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.055155 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
W20251029 08:52:25.057323 595 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58367 (request call id 154) took 1151 ms (client timeout 1000 ms). Trace:
W20251029 08:52:25.057494 595 rpcz_store.cc:269] 1029 08:52:23.905462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.910343 (+ 4881us) service_pool.cc:225] Handling call
1029 08:52:23.926350 (+ 16007us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb914ff2aa8 00007fb90bb05ca6 00007fb90bb059d5 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:25.057275 (+1130925us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":35136768}
I20251029 08:52:25.057701 596 catalog_manager.cc:5649] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:25.058249 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:25.064079 1974 rpcz_store.cc:276] 1029 08:52:22.564709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.564814 (+ 105us) service_pool.cc:225] Handling call
1029 08:52:25.037093 (+2472279us) ts_tablet_manager.cc:1151] Deleting tablet dbad8c4014b6423cb20e3e5c22ed69f1
1029 08:52:25.037103 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.037195 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2472076}
I20251029 08:52:25.043287 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "667c71a8dfd54c05ab56e845653c0c9a" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:25.064589 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet dbad8c4014b6423cb20e3e5c22ed69f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbad8c4014b6423cb20e3e5c22ed69f1 already in progress: creating tablet
W20251029 08:52:25.064664 996 leader_election.cc:343] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.057860 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 930314c8f90f462990f93fab8f7790c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.058740 889 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.059055 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.059216 892 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.065029 1974 tablet_replica.cc:333] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.062471 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:25.065202 1974 raft_consensus.cc:2243] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.065477 1974 raft_consensus.cc:2272] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.066102 1974 ts_tablet_manager.cc:1916] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.066974 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 78691bfecb0a41798096c27eecb4817f (DEFAULT_TABLE table=default.Table_c32c0cc51f75438f8786a435871e0c42 [id=6a338171500d4cff93155b8c6656bc6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.067498 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78691bfecb0a41798096c27eecb4817f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.068132 754 tablet_service.cc:1505] Processing CreateTablet for tablet 78691bfecb0a41798096c27eecb4817f (DEFAULT_TABLE table=default.Table_c32c0cc51f75438f8786a435871e0c42 [id=6a338171500d4cff93155b8c6656bc6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.068523 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78691bfecb0a41798096c27eecb4817f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.072757 892 tablet_service.cc:1505] Processing CreateTablet for tablet 78691bfecb0a41798096c27eecb4817f (DEFAULT_TABLE table=default.Table_c32c0cc51f75438f8786a435871e0c42 [id=6a338171500d4cff93155b8c6656bc6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.073127 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78691bfecb0a41798096c27eecb4817f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.080763 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_01f74d8d149e47da972673a83c50e16c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.081441 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f74d8d149e47da972673a83c50e16c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.082377 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7387b39af2fe4efa908305b60633960f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.082891 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7387b39af2fe4efa908305b60633960f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.083676 1183 tablet_bootstrap.cc:654] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.085938 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_064cacab68c74e94a8224b2c068aec83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.086452 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064cacab68c74e94a8224b2c068aec83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:25.090991 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a4ae5978b71443a48a88bc69128b187d (table default.Table_70588860077547cabad70977368b56fc [id=b07ebc517dd045a48caa07814f5c0580], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a4ae5978b71443a48a88bc69128b187d consensus state has no leader
I20251029 08:52:25.094470 2359 consensus_queue.cc:1048] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.095623 1045 tablet_service.cc:1505] Processing CreateTablet for tablet d300539872914cb086ad60b334d6d011 (DEFAULT_TABLE table=default.Table_871a2454944f4691b30571553cc9bbbb [id=b98cededcfc645ef9da00a9b9820f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.096181 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d300539872914cb086ad60b334d6d011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.096946 754 tablet_service.cc:1505] Processing CreateTablet for tablet d300539872914cb086ad60b334d6d011 (DEFAULT_TABLE table=default.Table_871a2454944f4691b30571553cc9bbbb [id=b98cededcfc645ef9da00a9b9820f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.097325 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d300539872914cb086ad60b334d6d011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.098465 892 tablet_service.cc:1505] Processing CreateTablet for tablet d300539872914cb086ad60b334d6d011 (DEFAULT_TABLE table=default.Table_871a2454944f4691b30571553cc9bbbb [id=b98cededcfc645ef9da00a9b9820f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.098841 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d300539872914cb086ad60b334d6d011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.100005 2554 consensus_queue.cc:1048] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.101343 1183 tablet_bootstrap.cc:492] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.101603 1183 ts_tablet_manager.cc:1403] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.003s
I20251029 08:52:25.102530 1183 raft_consensus.cc:359] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.102738 1183 raft_consensus.cc:385] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.102843 1183 raft_consensus.cc:740] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.103065 2485 consensus_queue.cc:1048] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:25.103271 861 consensus_peers.cc:597] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa411391c63146e7acb8b4c6edbe12a0. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.104763 2558 raft_consensus.cc:493] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.104872 2558 raft_consensus.cc:515] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.105456 2558 leader_election.cc:290] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.106959 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:25.107038 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85b300f17a704f7dba1aa3724fcc2de7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.107174 778 raft_consensus.cc:2393] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 927689e0dc5e4bc79d4a9d79a873f088 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:25.107340 2485 consensus_queue.cc:1048] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.108224 861 leader_election.cc:304] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.108548 2451 raft_consensus.cc:3060] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.109440 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5076743a16402388e401f94cbe6b3c (DEFAULT_TABLE table=default.Table_b71b3c0fa5504156ab80c2861bb9c67d [id=68dfef85ebce4d41acb292e2b6b92e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.109819 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5076743a16402388e401f94cbe6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.111248 2451 raft_consensus.cc:2749] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.119168 2485 raft_consensus.cc:493] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.119350 2485 raft_consensus.cc:515] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.120076 2485 leader_election.cc:290] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:25.121450 998 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:25.124198 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be4bcf9e3324bd884e946fae20648fd" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.124915 996 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.125075 996 leader_election.cc:304] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.128109 2359 raft_consensus.cc:2749] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.130085 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a2c13db40d854269b13340aa6acef83e
I20251029 08:52:25.130502 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8b1af4bcd7af49a0b8a419298ab37ea9
I20251029 08:52:25.130698 588 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39506 (request call id 1209) took 1014 ms. Trace:
I20251029 08:52:25.130985 590 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39506 (request call id 1201) took 1230 ms. Trace:
I20251029 08:52:25.131120 590 rpcz_store.cc:276] 1029 08:52:23.900043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.993180 (+ 93137us) service_pool.cc:225] Handling call
1029 08:52:24.009421 (+ 16241us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb914ff9256 00007fb90bb0a486 00007fb90bb0a1b5 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:24.009753 (+ 332us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1029 08:52:24.009878 (+ 125us) catalog_manager.cc:2815] Modifying in-memory table state
1029 08:52:24.009941 (+ 63us) catalog_manager.cc:2824] Locking tablets
1029 08:52:25.052271 (+1042330us) catalog_manager.cc:2838] Removing table and tablets from system table
1029 08:52:25.130295 (+ 78024us) catalog_manager.cc:2856] Removing table from by-name map
1029 08:52:25.130375 (+ 80us) catalog_manager.cc:2872] Committing in-memory state
1029 08:52:25.130425 (+ 50us) catalog_manager.cc:2876] Aborting table tasks
1029 08:52:25.130943 (+ 518us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.053587 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.053693 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.053698 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:25.053701 (+ 3us) tablet.cc:660] Decoding operations
1029 08:52:25.053810 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.053822 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.053826 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:25.053915 (+ 89us) tablet.cc:699] Row locks acquired
1029 08:52:25.053917 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:25.053955 (+ 38us) write_op.cc:270] Start()
1029 08:52:25.053991 (+ 36us) write_op.cc:276] Timestamp: P: 1761727945053911 usec, L: 0
1029 08:52:25.053994 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.087908 (+ 33914us) log.cc:844] Serialized 756 byte log entry
1029 08:52:25.088118 (+ 210us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb90d183862 00007fb90e5662a5 00007fb90e564dc9 00007fb90e56d1f0 00007fb90e56cffd 0000000000a8867c 00007fb901329d7f 00007fb90132d04b 00007fb90132cec8 0000000000a8867c
1029 08:52:25.128684 (+ 40566us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.129021 (+ 337us) write_op.cc:301] APPLY: starting
1029 08:52:25.129039 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.129087 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.129089 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.129159 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.129198 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 08:52:25.129205 (+ 7us) write_op.cc:312] APPLY: finished
1029 08:52:25.129283 (+ 78us) log.cc:844] Serialized 29 byte log entry
1029 08:52:25.129347 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.129365 (+ 18us) write_op.cc:454] Released schema lock
1029 08:52:25.129371 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:25.129455 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.129458 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":36833024,"child_traces":[["op",{"apply.queue_time_us":276,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":429,"num_ops":2,"prepare.queue_time_us":494,"prepare.run_cpu_time_us":1158,"prepare.run_wall_time_us":34575,"replication_time_us":74657,"spinlock_wait_cycles":77554432}]]}
I20251029 08:52:25.131737 758 tablet_service.cc:1552] Processing DeleteTablet for tablet a2c13db40d854269b13340aa6acef83e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.131954 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:25.132217 596 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:52653 (request call id 176) took 1193 ms (client timeout 1000 ms). Trace:
I20251029 08:52:25.132622 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:25.132325 596 rpcz_store.cc:269] 1029 08:52:23.938433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.989945 (+ 51512us) service_pool.cc:225] Handling call
1029 08:52:24.006126 (+ 16181us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb914ff2aa8 00007fb90bb05ca6 00007fb90bb059d5 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:25.132177 (+1126051us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.088853 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.088943 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.088948 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:25.088951 (+ 3us) tablet.cc:660] Decoding operations
1029 08:52:25.089022 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.089029 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.089034 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1029 08:52:25.089077 (+ 43us) tablet.cc:699] Row locks acquired
1029 08:52:25.089079 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:25.089104 (+ 25us) write_op.cc:270] Start()
1029 08:52:25.089126 (+ 22us) write_op.cc:276] Timestamp: P: 1761727945089059 usec, L: 0
1029 08:52:25.089129 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.089254 (+ 125us) log.cc:844] Serialized 527 byte log entry
1029 08:52:25.128804 (+ 39550us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.129934 (+ 1130us) write_op.cc:301] APPLY: starting
1029 08:52:25.129945 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.129970 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.130013 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.130045 (+ 32us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.130075 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:25.130082 (+ 7us) write_op.cc:312] APPLY: finished
1029 08:52:25.130128 (+ 46us) log.cc:844] Serialized 23 byte log entry
1029 08:52:25.130166 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.130176 (+ 10us) write_op.cc:454] Released schema lock
1029 08:52:25.130180 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":35904896,"child_traces":[["op",{"apply.queue_time_us":1099,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":302,"num_ops":1,"prepare.queue_time_us":29926,"prepare.run_cpu_time_us":459,"prepare.run_wall_time_us":458,"replication_time_us":39661,"threads_started":1}]]}
I20251029 08:52:25.130802 588 rpcz_store.cc:276] 1029 08:52:24.115953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.136439 (+ 20486us) service_pool.cc:225] Handling call
1029 08:52:24.136956 (+ 517us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1029 08:52:24.137136 (+ 180us) catalog_manager.cc:2815] Modifying in-memory table state
1029 08:52:24.137214 (+ 78us) catalog_manager.cc:2824] Locking tablets
1029 08:52:25.054617 (+917403us) catalog_manager.cc:2838] Removing table and tablets from system table
1029 08:52:25.129890 (+ 75273us) catalog_manager.cc:2856] Removing table from by-name map
1029 08:52:25.129983 (+ 93us) catalog_manager.cc:2872] Committing in-memory state
1029 08:52:25.130026 (+ 43us) catalog_manager.cc:2876] Aborting table tasks
1029 08:52:25.130628 (+ 602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.088198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.088294 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.088300 (+ 6us) write_op.cc:435] Acquired schema lock
1029 08:52:25.088303 (+ 3us) tablet.cc:660] Decoding operations
1029 08:52:25.088431 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.088443 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.088447 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:25.088534 (+ 87us) tablet.cc:699] Row locks acquired
1029 08:52:25.088537 (+ 3us) write_op.cc:260] PREPARE: finished
1029 08:52:25.088574 (+ 37us) write_op.cc:270] Start()
1029 08:52:25.088603 (+ 29us) write_op.cc:276] Timestamp: P: 1761727945088527 usec, L: 0
1029 08:52:25.088606 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.088757 (+ 151us) log.cc:844] Serialized 725 byte log entry
1029 08:52:25.128756 (+ 39999us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.129527 (+ 771us) write_op.cc:301] APPLY: starting
1029 08:52:25.129537 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.129567 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.129592 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.129647 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.129675 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 08:52:25.129683 (+ 8us) write_op.cc:312] APPLY: finished
1029 08:52:25.129736 (+ 53us) log.cc:844] Serialized 29 byte log entry
1029 08:52:25.129778 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.129791 (+ 13us) write_op.cc:454] Released schema lock
1029 08:52:25.129796 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:25.129865 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.129870 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":737,"apply.run_cpu_time_us":331,"apply.run_wall_time_us":331,"num_ops":2,"prepare.queue_time_us":32788,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":622,"replication_time_us":40132}]]}
I20251029 08:52:25.133553 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_83f92101d57a49da80bdc2552b39a1ba [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 1 got 0
I20251029 08:52:25.134131 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet af913080bfe44a86a76704ffa32cd9e1 (table default.Table_7fd53f3b59d14ad39715bc62e513ab79 [id=88bde60591764b37b719bdeea15ef0cd]). Expected version 1 got 0
I20251029 08:52:25.103084 1183 consensus_queue.cc:260] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.134881 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:25.135061 1183 ts_tablet_manager.cc:1434] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.033s user 0.000s sys 0.001s
I20251029 08:52:25.135165 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:25.135375 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]). Expected version 4 got 0
I20251029 08:52:25.135426 1183 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.136711 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a2c13db40d854269b13340aa6acef83e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.137706 594 catalog_manager.cc:5649] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:25.138005 1183 tablet_bootstrap.cc:654] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.138183 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:25.138468 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 026eef7a3cba40d0a2c090485cea55e0 (table default.Table_5471f52640fd473e90327f739bd8223d [id=559cd9144082414dade2ac5d73a7057c]). Expected version 2 got 0
I20251029 08:52:25.138725 594 catalog_manager.cc:5649] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:25.139175 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 5 got 0
I20251029 08:52:25.139497 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:25.142629 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a2c13db40d854269b13340aa6acef83e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.142869 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.147837 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5076743a16402388e401f94cbe6b3c (DEFAULT_TABLE table=default.Table_b71b3c0fa5504156ab80c2861bb9c67d [id=68dfef85ebce4d41acb292e2b6b92e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.148718 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5076743a16402388e401f94cbe6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.151082 2366 consensus_queue.cc:1048] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.158560 752 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5076743a16402388e401f94cbe6b3c (DEFAULT_TABLE table=default.Table_b71b3c0fa5504156ab80c2861bb9c67d [id=68dfef85ebce4d41acb292e2b6b92e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.159178 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5076743a16402388e401f94cbe6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.160293 2366 raft_consensus.cc:493] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.160470 2366 raft_consensus.cc:515] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.161186 2366 leader_election.cc:290] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.162014 1974 ts_tablet_manager.cc:1929] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.162139 1974 log.cc:1199] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/24ab9df6f768410eb4f66cffdcd68df2
W20251029 08:52:25.162181 998 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:25.162424 1974 ts_tablet_manager.cc:1950] T 24ab9df6f768410eb4f66cffdcd68df2 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.162341 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 980a0c3a08834c2484d7964d88f17bb0 (DEFAULT_TABLE table=default.Table_0fb0c73465d84a63805ee4f2b426b6de [id=3f493c7a0bd14940a7099e075258896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.162755 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980a0c3a08834c2484d7964d88f17bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.162752 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 955) took 2568 ms. Trace:
I20251029 08:52:25.162828 1974 rpcz_store.cc:276] 1029 08:52:22.594701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.594782 (+ 81us) service_pool.cc:225] Handling call
1029 08:52:25.064975 (+2470193us) ts_tablet_manager.cc:1151] Deleting tablet 24ab9df6f768410eb4f66cffdcd68df2
1029 08:52:25.064988 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.109454 (+ 44466us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.161891 (+ 52437us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.162735 (+ 844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470011}
I20251029 08:52:25.163275 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 24ab9df6f768410eb4f66cffdcd68df2 (table default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]) successfully deleted
I20251029 08:52:25.163625 1974 tablet_replica.cc:333] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.163807 1974 raft_consensus.cc:2243] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.163972 1974 raft_consensus.cc:2272] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.164551 1974 ts_tablet_manager.cc:1916] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.166852 888 tablet_service.cc:1505] Processing CreateTablet for tablet 980a0c3a08834c2484d7964d88f17bb0 (DEFAULT_TABLE table=default.Table_0fb0c73465d84a63805ee4f2b426b6de [id=3f493c7a0bd14940a7099e075258896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.167378 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980a0c3a08834c2484d7964d88f17bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.172247 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c4d57a59684723bfd27c7dda302ef3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.174592 2554 raft_consensus.cc:493] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.174790 2554 raft_consensus.cc:515] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.175664 2554 leader_election.cc:290] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.175791 1183 tablet_bootstrap.cc:492] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.175985 1183 ts_tablet_manager.cc:1403] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251029 08:52:25.176520 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39fc9612689b46d498d6092bd552da4e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.176785 1065 raft_consensus.cc:2468] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:25.178452 861 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 39fc9612689b46d498d6092bd552da4e
I20251029 08:52:25.178817 861 leader_election.cc:304] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.179479 2554 raft_consensus.cc:2804] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.179621 2554 raft_consensus.cc:493] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.179751 2554 raft_consensus.cc:3060] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.176857 1183 raft_consensus.cc:359] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.180469 1183 raft_consensus.cc:385] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.180608 1183 raft_consensus.cc:740] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.180881 1183 consensus_queue.cc:260] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.181490 1183 ts_tablet_manager.cc:1434] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:25.181592 1177 ts_tablet_manager.cc:1434] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.177s user 0.000s sys 0.001s
I20251029 08:52:25.181824 1183 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.182034 1177 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.183254 2554 raft_consensus.cc:515] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.183607 1183 tablet_bootstrap.cc:654] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.192533 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 254e62d88c3242a0a3ac7470a8005d74 (table default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]) successfully deleted
I20251029 08:52:25.193426 1176 tablet_replica.cc:333] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.193807 1176 raft_consensus.cc:2243] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.194052 1176 raft_consensus.cc:2272] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.195065 1176 ts_tablet_manager.cc:1916] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.197162 752 tablet_service.cc:1505] Processing CreateTablet for tablet 980a0c3a08834c2484d7964d88f17bb0 (DEFAULT_TABLE table=default.Table_0fb0c73465d84a63805ee4f2b426b6de [id=3f493c7a0bd14940a7099e075258896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.197808 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980a0c3a08834c2484d7964d88f17bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.200349 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 898) took 3045 ms. Trace:
I20251029 08:52:25.200546 1179 rpcz_store.cc:276] 1029 08:52:22.155191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.155903 (+ 712us) service_pool.cc:225] Handling call
1029 08:52:25.200191 (+3044288us) ts_tablet_manager.cc:1151] Deleting tablet 63e705a0b5264eb7a6988129edbf5970
1029 08:52:25.200204 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.200328 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3044088}
W20251029 08:52:25.201157 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 63e705a0b5264eb7a6988129edbf5970 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e705a0b5264eb7a6988129edbf5970 already in progress: creating tablet
I20251029 08:52:25.201253 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 895) took 3046 ms. Trace:
I20251029 08:52:25.201663 1179 rpcz_store.cc:276] 1029 08:52:22.154898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.155246 (+ 348us) service_pool.cc:225] Handling call
1029 08:52:25.201119 (+3045873us) ts_tablet_manager.cc:1151] Deleting tablet aa348d6fa8c644bd8fe7b297a53c1251
1029 08:52:25.201129 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.201221 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3044561}
I20251029 08:52:25.202216 1179 tablet_replica.cc:333] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:25.202239 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet aa348d6fa8c644bd8fe7b297a53c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa348d6fa8c644bd8fe7b297a53c1251 already in progress: creating tablet
I20251029 08:52:25.202549 1179 raft_consensus.cc:2243] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.202786 1179 raft_consensus.cc:2272] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.208245 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 8a5bcee728df4ab68d467f515fe1fafd (DEFAULT_TABLE table=default.Table_0f0ca950566740a49562ebfa6fb1772a [id=e1066ec599a049fbaa3385a56145dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.208735 2458 raft_consensus.cc:493] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.208865 2458 raft_consensus.cc:515] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.209530 2458 leader_election.cc:290] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.209488 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 3285b8fc51b24255b195b019e6b731d4 (DEFAULT_TABLE table=default.Table_2ca9e5a740e943df8671c43f1e17dc05 [id=32f19bef7717461e8ebb6d55b6b2d353]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.209899 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3285b8fc51b24255b195b019e6b731d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.210163 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5bcee728df4ab68d467f515fe1fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.210258 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.210806 720 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.212280 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e2dbaf0f3d3242b388197299431cf86d (DEFAULT_TABLE table=default.Table_98da77093ed14699a141e885b30e3088 [id=ad1cbf16321a496297d5c3eb96943458]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.212633 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbaf0f3d3242b388197299431cf86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.214519 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f062556d67a148d19d242b8d99cfdff9
I20251029 08:52:25.215209 597 master_service.cc:954] ReplaceTablet: received request to replace tablet d0d00a6b772d4034ac7fa764884c7081 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:25.216611 758 tablet_service.cc:1505] Processing CreateTablet for tablet 3285b8fc51b24255b195b019e6b731d4 (DEFAULT_TABLE table=default.Table_2ca9e5a740e943df8671c43f1e17dc05 [id=32f19bef7717461e8ebb6d55b6b2d353]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.216801 2458 raft_consensus.cc:493] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.216924 2458 raft_consensus.cc:515] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.217027 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3285b8fc51b24255b195b019e6b731d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.217550 2458 leader_election.cc:290] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.218611 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.219172 720 leader_election.cc:343] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.219770 753 tablet_service.cc:1552] Processing DeleteTablet for tablet f062556d67a148d19d242b8d99cfdff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.219787 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet f062556d67a148d19d242b8d99cfdff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.220203 748 tablet_service.cc:1505] Processing CreateTablet for tablet e2dbaf0f3d3242b388197299431cf86d (DEFAULT_TABLE table=default.Table_98da77093ed14699a141e885b30e3088 [id=ad1cbf16321a496297d5c3eb96943458]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.220590 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbaf0f3d3242b388197299431cf86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.174260 919 raft_consensus.cc:2468] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:25.224314 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.224511 996 leader_election.cc:304] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:25.228447 718 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.228607 718 leader_election.cc:304] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.228904 2458 raft_consensus.cc:2749] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.229475 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8a5bcee728df4ab68d467f515fe1fafd (DEFAULT_TABLE table=default.Table_0f0ca950566740a49562ebfa6fb1772a [id=e1066ec599a049fbaa3385a56145dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.230142 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5bcee728df4ab68d467f515fe1fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.237704 2485 raft_consensus.cc:2804] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.237847 2485 raft_consensus.cc:493] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.237921 2485 raft_consensus.cc:3060] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.240785 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39fc9612689b46d498d6092bd552da4e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.241020 1065 raft_consensus.cc:3060] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:25.242568 861 leader_election.cc:343] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 39fc9612689b46d498d6092bd552da4e
I20251029 08:52:25.242651 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_45ce89ee1267486fb1dc5d6990cb0716" } new_table_name: "default.Table_cdae86f1af1840569f67820138d741ea" modify_external_catalogs: true
I20251029 08:52:25.244127 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3676333781d94b0d80010190fd318da8" } new_table_name: "default.Table_4d8742cba3e14ea3b056d42485790edc" modify_external_catalogs: true
I20251029 08:52:25.244261 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f3b73a492eca4e8fa1f1365cbc3518af" } new_table_name: "default.Table_4f7ad1f9e93c45f782618995a8dda857" modify_external_catalogs: true
I20251029 08:52:25.245625 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_84ecf561646e4d788d12eccac9cb8646"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.246191 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84ecf561646e4d788d12eccac9cb8646 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.248278 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c2392d03cd0b46e08f84a4eee1831855" } new_table_name: "default.Table_60a71e471a9f4523a163663b4edbf15d" modify_external_catalogs: true
I20251029 08:52:25.249529 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_122ac766040a47f39ba5012bc56c5029" } new_table_name: "default.Table_5fd1636a22104e95a64e29baa9584e3c" modify_external_catalogs: true
I20251029 08:52:25.254323 2485 raft_consensus.cc:515] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.255506 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8c4d57a59684723bfd27c7dda302ef3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:25.255740 922 raft_consensus.cc:3060] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.257908 1065 raft_consensus.cc:2468] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.258373 2485 leader_election.cc:290] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.258440 861 leader_election.cc:304] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.258639 922 raft_consensus.cc:2468] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.258780 2451 raft_consensus.cc:2804] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.258894 2451 raft_consensus.cc:697] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:25.259083 996 leader_election.cc:304] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:25.259140 2451 consensus_queue.cc:237] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.259372 2485 raft_consensus.cc:2804] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.259500 2485 raft_consensus.cc:697] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.259770 2485 consensus_queue.cc:237] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.261782 998 leader_election.cc:343] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:25.269191 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:25.269448 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:25.269719 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:25.270056 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 5 got 0
I20251029 08:52:25.270774 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 69d27944f3c14a6eb8331a904d6faa40 (table default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654]). Expected version 1 got 0
I20251029 08:52:25.183811 1177 tablet_bootstrap.cc:654] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.206890 1974 ts_tablet_manager.cc:1929] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.207796 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8a5bcee728df4ab68d467f515fe1fafd (DEFAULT_TABLE table=default.Table_0f0ca950566740a49562ebfa6fb1772a [id=e1066ec599a049fbaa3385a56145dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.272239 1974 log.cc:1199] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/fe384ca428974ac5ba78e6506a1a8832
I20251029 08:52:25.272545 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5bcee728df4ab68d467f515fe1fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.272614 1974 ts_tablet_manager.cc:1950] T fe384ca428974ac5ba78e6506a1a8832 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.272908 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 956) took 2677 ms. Trace:
I20251029 08:52:25.272979 1974 rpcz_store.cc:276] 1029 08:52:22.595078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.595246 (+ 168us) service_pool.cc:225] Handling call
1029 08:52:25.163575 (+2568329us) ts_tablet_manager.cc:1151] Deleting tablet fe384ca428974ac5ba78e6506a1a8832
1029 08:52:25.163586 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.191501 (+ 27915us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.206769 (+ 15268us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.272893 (+ 66124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568177}
I20251029 08:52:25.273293 1974 tablet_replica.cc:333] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.273515 1974 raft_consensus.cc:2243] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.273806 1974 raft_consensus.cc:2272] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.274225 2366 consensus_queue.cc:1048] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.274667 1974 ts_tablet_manager.cc:1916] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.209949 890 tablet_service.cc:1505] Processing CreateTablet for tablet 3285b8fc51b24255b195b019e6b731d4 (DEFAULT_TABLE table=default.Table_2ca9e5a740e943df8671c43f1e17dc05 [id=32f19bef7717461e8ebb6d55b6b2d353]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.276351 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3285b8fc51b24255b195b019e6b731d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.212673 887 tablet_service.cc:1505] Processing CreateTablet for tablet e2dbaf0f3d3242b388197299431cf86d (DEFAULT_TABLE table=default.Table_98da77093ed14699a141e885b30e3088 [id=ad1cbf16321a496297d5c3eb96943458]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.216491 896 tablet_service.cc:1552] Processing DeleteTablet for tablet f062556d67a148d19d242b8d99cfdff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.218428 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.279865 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbaf0f3d3242b388197299431cf86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.282274 1183 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.282459 1183 ts_tablet_manager.cc:1403] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.000s
I20251029 08:52:25.283531 1183 raft_consensus.cc:359] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.283749 1183 raft_consensus.cc:385] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.283829 1183 raft_consensus.cc:740] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.284080 1183 consensus_queue.cc:260] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.284570 1183 ts_tablet_manager.cc:1434] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:25.284885 1183 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:25.284973 718 leader_election.cc:343] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.285116 718 leader_election.cc:304] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.285694 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet fe384ca428974ac5ba78e6506a1a8832 (table default.Table_208a79319e20472782aba43b998349ea [id=b6f56ab1175642b0b0d8b19735f62bc6]) successfully deleted
I20251029 08:52:25.286172 2458 raft_consensus.cc:2749] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.290256 1178 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.290457 1178 ts_tablet_manager.cc:1403] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.245s user 0.005s sys 0.020s
I20251029 08:52:25.291215 1178 raft_consensus.cc:359] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.291440 1178 raft_consensus.cc:385] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.291531 1178 raft_consensus.cc:740] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.291913 1178 consensus_queue.cc:260] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.292564 1178 ts_tablet_manager.cc:1434] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:25.292886 1178 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.295207 1183 tablet_bootstrap.cc:654] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.299206 1178 tablet_bootstrap.cc:654] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:25.299293 594 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:60007 (request call id 152) took 825 ms (client timeout 1000 ms). Trace:
W20251029 08:52:25.299453 594 rpcz_store.cc:269] 1029 08:52:24.474245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.508449 (+ 34204us) service_pool.cc:225] Handling call
1029 08:52:25.299245 (+790796us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:25.242972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:25.243109 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:25.243116 (+ 7us) write_op.cc:435] Acquired schema lock
1029 08:52:25.243118 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:25.243239 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:25.243252 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:25.243256 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 08:52:25.243392 (+ 136us) tablet.cc:699] Row locks acquired
1029 08:52:25.243394 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:25.243431 (+ 37us) write_op.cc:270] Start()
1029 08:52:25.243465 (+ 34us) write_op.cc:276] Timestamp: P: 1761727945243368 usec, L: 0
1029 08:52:25.243469 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 08:52:25.243631 (+ 162us) log.cc:844] Serialized 901 byte log entry
1029 08:52:25.257467 (+ 13836us) op_driver.cc:464] REPLICATION: finished
1029 08:52:25.263825 (+ 6358us) write_op.cc:301] APPLY: starting
1029 08:52:25.263846 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:25.263896 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:25.263924 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:25.264029 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:25.264071 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 08:52:25.264078 (+ 7us) write_op.cc:312] APPLY: finished
1029 08:52:25.264169 (+ 91us) log.cc:844] Serialized 29 byte log entry
1029 08:52:25.265304 (+ 1135us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:25.265330 (+ 26us) write_op.cc:454] Released schema lock
1029 08:52:25.265336 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":6316,"num_ops":2,"prepare.queue_time_us":98081,"prepare.run_cpu_time_us":6596,"prepare.run_wall_time_us":45693,"raft.queue_time_us":8444,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":13982,"thread_start_us":44909,"threads_started":1}]]}
I20251029 08:52:25.302188 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.302476 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.302773 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.303061 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_2073f0e27ae9417eb3f4273468277ff0 [id=38aeb2e83ccc408aba7f1a5438c2f6fa] alter complete (version 1)
I20251029 08:52:25.303205 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet bd0c43367a8c47f390cc61d2764d61e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.305356 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.306216 1179 ts_tablet_manager.cc:1916] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.313319 1974 ts_tablet_manager.cc:1929] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.313504 1974 log.cc:1199] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aafa846cb2144cec9ce4b28e61d91961
I20251029 08:52:25.313902 1974 ts_tablet_manager.cc:1950] T aafa846cb2144cec9ce4b28e61d91961 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.314285 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 954) took 2720 ms. Trace:
I20251029 08:52:25.314419 1974 rpcz_store.cc:276] 1029 08:52:22.593940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.607791 (+ 13851us) service_pool.cc:225] Handling call
1029 08:52:25.273253 (+2665462us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:25.273262 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.277943 (+ 4681us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.313222 (+ 35279us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.314265 (+ 1043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2665304}
I20251029 08:52:25.314992 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet aafa846cb2144cec9ce4b28e61d91961 (table default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]) successfully deleted
I20251029 08:52:25.315490 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 957) took 2662 ms. Trace:
I20251029 08:52:25.315609 1974 rpcz_store.cc:276] 1029 08:52:22.653478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.672351 (+ 18873us) service_pool.cc:225] Handling call
1029 08:52:25.315332 (+2642981us) ts_tablet_manager.cc:1151] Deleting tablet a186df0db24c4d22abee7ec57bcb199b
1029 08:52:25.315346 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.315471 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2642766}
W20251029 08:52:25.316128 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a186df0db24c4d22abee7ec57bcb199b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a186df0db24c4d22abee7ec57bcb199b already in progress: creating tablet
I20251029 08:52:25.316519 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 958) took 2539 ms. Trace:
I20251029 08:52:25.316622 1974 rpcz_store.cc:276] 1029 08:52:22.776664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.777460 (+ 796us) service_pool.cc:225] Handling call
1029 08:52:25.316408 (+2538948us) ts_tablet_manager.cc:1151] Deleting tablet 5c93076ca44542a5ae19e2c73dba95cb
1029 08:52:25.316417 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.316502 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2538615}
W20251029 08:52:25.317018 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5c93076ca44542a5ae19e2c73dba95cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c93076ca44542a5ae19e2c73dba95cb already in progress: creating tablet
I20251029 08:52:25.317297 1974 tablet_replica.cc:333] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.317780 1974 raft_consensus.cc:2243] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.318542 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 61b07eb4befa41f3a1b5f2e9ebc2277c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:25.320140 1178 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.327334 1178 ts_tablet_manager.cc:1403] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20251029 08:52:25.328840 1178 raft_consensus.cc:359] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.329164 1178 raft_consensus.cc:385] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.329277 1178 raft_consensus.cc:740] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:25.329546 996 consensus_peers.cc:597] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aafa846cb2144cec9ce4b28e61d91961. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.329571 1178 consensus_queue.cc:260] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.330183 1178 ts_tablet_manager.cc:1434] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:25.330515 1178 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.332585 1183 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.332907 2475 raft_consensus.cc:493] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.333047 1183 ts_tablet_manager.cc:1403] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.002s
I20251029 08:52:25.333073 2475 raft_consensus.cc:515] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.334064 1183 raft_consensus.cc:359] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.334285 1183 raft_consensus.cc:385] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.334429 1183 raft_consensus.cc:740] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.334510 2475 leader_election.cc:290] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.334710 1183 consensus_queue.cc:260] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.335296 998 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:25.321525 2554 leader_election.cc:290] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.327170 1177 tablet_bootstrap.cc:492] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.335767 1177 ts_tablet_manager.cc:1403] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.154s user 0.003s sys 0.000s
I20251029 08:52:25.335703 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f5ebd2bf6614980ac0c18f5d9974fc0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.336012 924 raft_consensus.cc:2468] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:25.336302 2458 raft_consensus.cc:493] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.336416 2458 raft_consensus.cc:515] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.336968 2458 leader_election.cc:290] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.337446 996 leader_election.cc:304] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.337906 2485 raft_consensus.cc:2804] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.338013 2485 raft_consensus.cc:493] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.338097 2485 raft_consensus.cc:3060] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.341137 2485 raft_consensus.cc:515] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.341979 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f5ebd2bf6614980ac0c18f5d9974fc0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:25.342139 924 raft_consensus.cc:3060] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.343253 758 tablet_service.cc:1505] Processing CreateTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b (DEFAULT_TABLE table=default.Table_bb3e353385b14cfaa0d01935be38e1e9 [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:25.345001 998 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:25.346118 924 raft_consensus.cc:2468] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.346596 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8348b6234e4dbe8bdb58edad34ea7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.347005 889 tablet_service.cc:1505] Processing CreateTablet for tablet 46e839acfe394a32ab0b51af1dea253e (DEFAULT_TABLE table=default.Table_3ced4474768d425dae572ad46c65317c [id=e7423ecae55d4f5da5e81dd2db39c54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.349306 1176 ts_tablet_manager.cc:1929] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.349395 1176 log.cc:1199] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4e213713757949caa3f504f70026b7b5
I20251029 08:52:25.349668 1176 ts_tablet_manager.cc:1950] T 4e213713757949caa3f504f70026b7b5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.352005 753 tablet_service.cc:1505] Processing CreateTablet for tablet 46e839acfe394a32ab0b51af1dea253e (DEFAULT_TABLE table=default.Table_3ced4474768d425dae572ad46c65317c [id=e7423ecae55d4f5da5e81dd2db39c54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.352389 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e839acfe394a32ab0b51af1dea253e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.354141 996 leader_election.cc:304] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.355273 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 46e839acfe394a32ab0b51af1dea253e (DEFAULT_TABLE table=default.Table_3ced4474768d425dae572ad46c65317c [id=e7423ecae55d4f5da5e81dd2db39c54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.355640 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e839acfe394a32ab0b51af1dea253e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.360653 2485 leader_election.cc:290] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.360966 2485 raft_consensus.cc:2804] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.361079 2485 raft_consensus.cc:697] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.361320 2485 consensus_queue.cc:237] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.362040 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70b8c594aa24f37a169d1aa29c2268e consensus state has no leader
W20251029 08:52:25.362272 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b0f1cdfb1804d1c8d6220907795fc80 consensus state has no leader
I20251029 08:52:25.365557 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b (DEFAULT_TABLE table=default.Table_bb3e353385b14cfaa0d01935be38e1e9 [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.365921 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8348b6234e4dbe8bdb58edad34ea7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:25.366513 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4cb38b4023242fa8e85356dd4cccd70 consensus state has no leader
I20251029 08:52:25.369675 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 (DEFAULT_TABLE table=default.Table_038f3279c2364bdeb2dc381a25b699b5 [id=6675f26900cb4caf98d72b60fff9b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.370054 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a85a1dff3034284a6dcf0e8c3f53297. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.371228 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 911) took 3352 ms. Trace:
I20251029 08:52:25.371316 1176 rpcz_store.cc:276] 1029 08:52:22.019072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.028267 (+ 9195us) service_pool.cc:225] Handling call
1029 08:52:25.193176 (+3164909us) ts_tablet_manager.cc:1151] Deleting tablet 4e213713757949caa3f504f70026b7b5
1029 08:52:25.193194 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.327982 (+134788us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.349219 (+ 21237us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.371203 (+ 21984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3164697}
I20251029 08:52:25.371738 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4e213713757949caa3f504f70026b7b5 (table default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]) successfully deleted
I20251029 08:52:25.373061 1176 tablet_replica.cc:333] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:25.374830 996 consensus_peers.cc:597] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24ab9df6f768410eb4f66cffdcd68df2. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.376295 1176 raft_consensus.cc:2243] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.376747 1176 raft_consensus.cc:2272] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.377502 1176 ts_tablet_manager.cc:1916] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.378922 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.379148 1067 raft_consensus.cc:2468] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:25.379509 1179 ts_tablet_manager.cc:1929] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.379611 1179 log.cc:1199] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c66cbb495f5142d79a674f7d5adc505e
I20251029 08:52:25.379626 720 leader_election.cc:304] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:25.379876 1179 ts_tablet_manager.cc:1950] T c66cbb495f5142d79a674f7d5adc505e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:25.379879 2458 raft_consensus.cc:2804] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.379940 2458 raft_consensus.cc:493] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.379994 2458 raft_consensus.cc:3060] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.382982 2458 raft_consensus.cc:515] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.384537 2458 leader_election.cc:290] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.385473 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.385707 1067 raft_consensus.cc:3060] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.388358 888 tablet_service.cc:1505] Processing CreateTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b (DEFAULT_TABLE table=default.Table_bb3e353385b14cfaa0d01935be38e1e9 [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.388716 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.389279 718 leader_election.cc:343] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.390215 1067 raft_consensus.cc:2468] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:25.390825 720 leader_election.cc:304] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:25.391291 2545 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:25.391374 2458 raft_consensus.cc:2804] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.391582 2458 raft_consensus.cc:697] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:25.391816 2458 consensus_queue.cc:237] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.392969 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:25.396886 718 leader_election.cc:343] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.399977 890 tablet_service.cc:1505] Processing CreateTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 (DEFAULT_TABLE table=default.Table_038f3279c2364bdeb2dc381a25b699b5 [id=6675f26900cb4caf98d72b60fff9b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.402005 758 tablet_service.cc:1505] Processing CreateTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 (DEFAULT_TABLE table=default.Table_038f3279c2364bdeb2dc381a25b699b5 [id=6675f26900cb4caf98d72b60fff9b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.402487 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a85a1dff3034284a6dcf0e8c3f53297. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.403389 2458 raft_consensus.cc:493] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.403519 2458 raft_consensus.cc:515] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.404415 2458 leader_election.cc:290] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.405030 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.405718 720 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.406144 1176 ts_tablet_manager.cc:1929] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.406250 1176 log.cc:1199] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7a355c68ecfb45ff8c396873d91268a8
I20251029 08:52:25.406574 1176 ts_tablet_manager.cc:1950] T 7a355c68ecfb45ff8c396873d91268a8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.407043 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.407290 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05 (DEFAULT_TABLE table=default.Table_78669d09485b482a90061cad43238368 [id=d1302f2cde874c3fb4a6612d9213952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.408192 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:25.407510 718 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.409080 718 leader_election.cc:304] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.409454 2458 raft_consensus.cc:2749] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.411782 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 910) took 3392 ms. Trace:
I20251029 08:52:25.412523 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e839acfe394a32ab0b51af1dea253e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.415465 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05 (DEFAULT_TABLE table=default.Table_78669d09485b482a90061cad43238368 [id=d1302f2cde874c3fb4a6612d9213952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.416543 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.419716 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f8348b6234e4dbe8bdb58edad34ea7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.420025 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a85a1dff3034284a6dcf0e8c3f53297. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:25.420733 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bae8ba92b214a7c99bec970f3cfb68d consensus state has no leader
I20251029 08:52:25.421159 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3dbf386f200442c097b8998936081933"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.421741 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dbf386f200442c097b8998936081933 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.422308 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66eb18b4419b42f087e8468a9dc15153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.422837 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66eb18b4419b42f087e8468a9dc15153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.423269 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6917eb552d404f19b2e477c77b0fcf51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.423704 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6917eb552d404f19b2e477c77b0fcf51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.425560 1177 raft_consensus.cc:359] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.426867 1177 raft_consensus.cc:385] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.427070 1177 raft_consensus.cc:740] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.428284 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ac500003d4a84f739ff1c0ac8d40e93c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.428799 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac500003d4a84f739ff1c0ac8d40e93c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.430195 1176 rpcz_store.cc:276] 1029 08:52:22.018981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.030291 (+ 11310us) service_pool.cc:225] Handling call
1029 08:52:25.373007 (+3342716us) ts_tablet_manager.cc:1151] Deleting tablet 7a355c68ecfb45ff8c396873d91268a8
1029 08:52:25.373018 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.387646 (+ 14628us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.406042 (+ 18396us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.411764 (+ 5722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342526}
I20251029 08:52:25.430696 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 920) took 3389 ms. Trace:
I20251029 08:52:25.430784 1176 rpcz_store.cc:276] 1029 08:52:22.041558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.041613 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:25.430515 (+3388902us) ts_tablet_manager.cc:1151] Deleting tablet aa348d6fa8c644bd8fe7b297a53c1251
1029 08:52:25.430529 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.430673 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3388685}
I20251029 08:52:25.430856 758 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05 (DEFAULT_TABLE table=default.Table_78669d09485b482a90061cad43238368 [id=d1302f2cde874c3fb4a6612d9213952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.431234 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0638cbf1bc4c4cbd3dcad866ca0a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.431931 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7a355c68ecfb45ff8c396873d91268a8 (table default.Table_a3dc21093f304d13933e68c784dbfe5e [id=91274c3e8fac4bdbbbffefa10cd73044]) successfully deleted
I20251029 08:52:25.432158 1177 consensus_queue.cc:260] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.432649 1177 ts_tablet_manager.cc:1434] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.097s user 0.001s sys 0.000s
I20251029 08:52:25.432933 1177 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.434110 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:25.434437 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
W20251029 08:52:25.434635 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a52a754aca0467d9733107685a3bca2 consensus state has no leader
I20251029 08:52:25.434742 1183 ts_tablet_manager.cc:1434] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:25.435058 1183 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.435055 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:25.435017 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_01d382774b104ee89690fbab1a407cab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:25.435233 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
W20251029 08:52:25.435513 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01d382774b104ee89690fbab1a407cab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.436645 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d0d00a6b772d4034ac7fa764884c7081
I20251029 08:52:25.437173 597 catalog_manager.cc:6719] ReplaceTablet: tablet d0d00a6b772d4034ac7fa764884c7081 deleted and replaced by tablet e57e4313adce4e35843ccd9ee137e8f7
I20251029 08:52:25.437427 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 90360621ce124475af2f12060471f63c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:25.437589 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d0d00a6b772d4034ac7fa764884c7081 with delete_type TABLET_DATA_DELETED (replaced by tablet e57e4313adce4e35843ccd9ee137e8f7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.437671 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d0d00a6b772d4034ac7fa764884c7081 with delete_type TABLET_DATA_DELETED (replaced by tablet e57e4313adce4e35843ccd9ee137e8f7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.437947 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_9cb7ad5e75534b2897602246ecb55a74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.438426 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cb7ad5e75534b2897602246ecb55a74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.440057 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d0d00a6b772d4034ac7fa764884c7081 with delete_type TABLET_DATA_DELETED (replaced by tablet e57e4313adce4e35843ccd9ee137e8f7) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.435531 596 catalog_manager.cc:5649] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:25.440904 1974 raft_consensus.cc:2272] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.441061 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 026eef7a3cba40d0a2c090485cea55e0 (table default.Table_5471f52640fd473e90327f739bd8223d [id=559cd9144082414dade2ac5d73a7057c]). Expected version 2 got 0
I20251029 08:52:25.441306 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 85b300f17a704f7dba1aa3724fcc2de7 (table default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2]). Expected version 5 got 0
I20251029 08:52:25.442134 1178 tablet_bootstrap.cc:654] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.443310 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 2f816afb0d084c6db44be8a41f53ce04 (DEFAULT_TABLE table=default.Table_21d20ee24eca44568b6a69044d5c55a4 [id=436174f0eefd441aa0719ec72739aba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.443745 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f816afb0d084c6db44be8a41f53ce04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.445856 1974 ts_tablet_manager.cc:1916] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.447964 1183 tablet_bootstrap.cc:654] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.448325 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2f816afb0d084c6db44be8a41f53ce04 (DEFAULT_TABLE table=default.Table_21d20ee24eca44568b6a69044d5c55a4 [id=436174f0eefd441aa0719ec72739aba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.448738 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f816afb0d084c6db44be8a41f53ce04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.448860 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_65ece294edec4bf395c6d737cfb14e29 [id=5dc0a7f470cb432dbd918d84b96dd58b] alter complete (version 3)
I20251029 08:52:25.449625 2554 raft_consensus.cc:493] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.449852 2554 raft_consensus.cc:515] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.450433 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bceccce7081b43f6ad2849ed269dfe73" } modify_external_catalogs: true
I20251029 08:52:25.450630 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bceccce7081b43f6ad2849ed269dfe73" } modify_external_catalogs: true
I20251029 08:52:25.451467 1177 tablet_bootstrap.cc:654] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.452324 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:25.452554 783 raft_consensus.cc:2468] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:25.453006 861 leader_election.cc:304] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:25.453074 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_936021e20d444b0e9067bb236b20bae2" } modify_external_catalogs: true
I20251029 08:52:25.453218 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_936021e20d444b0e9067bb236b20bae2" } modify_external_catalogs: true
I20251029 08:52:25.451243 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.453473 1067 raft_consensus.cc:2468] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:25.453970 2554 leader_election.cc:290] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.454465 2554 raft_consensus.cc:2804] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.454643 2554 raft_consensus.cc:493] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.454854 2554 raft_consensus.cc:3060] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.455307 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 5498826a9afd4fd5a6b88b0a465fff40 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:25.463209 2545 tablet.cc:1722] T 69d27944f3c14a6eb8331a904d6faa40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:25.464282 2609 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:25.467891 890 tablet_service.cc:1505] Processing CreateTablet for tablet 3022a74550e545f49bcff1437a5cf9a5 (DEFAULT_TABLE table=default.Table_de611c4a46854ca195eb3cd4001470bf [id=89f6c5d6462e42fe8eb67ff8fea28054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.468150 2485 raft_consensus.cc:493] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.468185 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:25.468283 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3022a74550e545f49bcff1437a5cf9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.468374 2485 raft_consensus.cc:515] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.468667 594 catalog_manager.cc:6719] ReplaceTablet: tablet 61b07eb4befa41f3a1b5f2e9ebc2277c deleted and replaced by tablet d51192ae278d46d4b169e2ffd76e9d7c
I20251029 08:52:25.468889 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c with delete_type TABLET_DATA_DELETED (replaced by tablet d51192ae278d46d4b169e2ffd76e9d7c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.468991 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_86364f11715846199b42645e6deeb2fb" } modify_external_catalogs: true
I20251029 08:52:25.469069 2485 leader_election.cc:290] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.469139 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_86364f11715846199b42645e6deeb2fb" } modify_external_catalogs: true
W20251029 08:52:25.470314 998 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:25.471118 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c with delete_type TABLET_DATA_DELETED (replaced by tablet d51192ae278d46d4b169e2ffd76e9d7c) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:25.471724 861 consensus_peers.cc:597] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd0c43367a8c47f390cc61d2764d61e4. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.472718 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c with delete_type TABLET_DATA_DELETED (replaced by tablet d51192ae278d46d4b169e2ffd76e9d7c) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:25.473351 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet aa348d6fa8c644bd8fe7b297a53c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa348d6fa8c644bd8fe7b297a53c1251 already in progress: creating tablet
I20251029 08:52:25.473825 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 921) took 3419 ms. Trace:
I20251029 08:52:25.474108 2603 tablet.cc:1722] T 69d27944f3c14a6eb8331a904d6faa40 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:25.474324 1176 rpcz_store.cc:276] 1029 08:52:22.054824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.055439 (+ 615us) service_pool.cc:225] Handling call
1029 08:52:25.473710 (+3418271us) ts_tablet_manager.cc:1151] Deleting tablet b7a6008b4e6243ccaf166a31f1a9f62c
1029 08:52:25.473719 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.473809 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418052}
I20251029 08:52:25.474751 2602 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 08:52:25.474856 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c1a8a1e4cc14f1aa6e4f91174f45127" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.480418 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b7a6008b4e6243ccaf166a31f1a9f62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6008b4e6243ccaf166a31f1a9f62c already in progress: creating tablet
I20251029 08:52:25.480955 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 922) took 3348 ms. Trace:
I20251029 08:52:25.481313 1177 tablet_bootstrap.cc:492] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.481353 1176 rpcz_store.cc:276] 1029 08:52:22.132613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.132787 (+ 174us) service_pool.cc:225] Handling call
1029 08:52:25.480835 (+3348048us) ts_tablet_manager.cc:1151] Deleting tablet 63e705a0b5264eb7a6988129edbf5970
1029 08:52:25.480844 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.480939 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3347834}
I20251029 08:52:25.481747 1177 ts_tablet_manager.cc:1403] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
W20251029 08:52:25.481753 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 63e705a0b5264eb7a6988129edbf5970 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e705a0b5264eb7a6988129edbf5970 already in progress: creating tablet
I20251029 08:52:25.482264 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 923) took 3245 ms. Trace:
I20251029 08:52:25.482338 1176 rpcz_store.cc:276] 1029 08:52:22.236547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.236613 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:25.482170 (+3245557us) ts_tablet_manager.cc:1151] Deleting tablet b582dfeab662456ba4730f5c80455d83
1029 08:52:25.482178 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.482251 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3245347}
I20251029 08:52:25.482586 1177 raft_consensus.cc:359] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:25.482717 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b582dfeab662456ba4730f5c80455d83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b582dfeab662456ba4730f5c80455d83 already in progress: creating tablet
I20251029 08:52:25.482834 1177 raft_consensus.cc:385] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.482921 1177 raft_consensus.cc:740] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.483162 1177 consensus_queue.cc:260] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.483439 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 924) took 3211 ms. Trace:
I20251029 08:52:25.483513 1176 rpcz_store.cc:276] 1029 08:52:22.271571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.271626 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:25.483320 (+3211694us) ts_tablet_manager.cc:1151] Deleting tablet 0b61c710897f43f798d88d92c4676d0d
1029 08:52:25.483329 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.483425 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211495}
W20251029 08:52:25.483839 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0b61c710897f43f798d88d92c4676d0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b61c710897f43f798d88d92c4676d0d already in progress: creating tablet
I20251029 08:52:25.483862 1177 ts_tablet_manager.cc:1434] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:25.476995 758 tablet_service.cc:1505] Processing CreateTablet for tablet 2f816afb0d084c6db44be8a41f53ce04 (DEFAULT_TABLE table=default.Table_21d20ee24eca44568b6a69044d5c55a4 [id=436174f0eefd441aa0719ec72739aba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.484215 1177 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.484361 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f816afb0d084c6db44be8a41f53ce04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.484549 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 929) took 3109 ms. Trace:
I20251029 08:52:25.484920 1176 rpcz_store.cc:276] 1029 08:52:22.375347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.375393 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:25.484452 (+3109059us) ts_tablet_manager.cc:1151] Deleting tablet d09ed524e97c4c1bb3c4aaea876415ea
1029 08:52:25.484461 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.484533 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108867}
W20251029 08:52:25.485502 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d09ed524e97c4c1bb3c4aaea876415ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d09ed524e97c4c1bb3c4aaea876415ea already in progress: creating tablet
I20251029 08:52:25.485905 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 928) took 3110 ms. Trace:
I20251029 08:52:25.486259 1176 rpcz_store.cc:276] 1029 08:52:22.375192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.382450 (+ 7258us) service_pool.cc:225] Handling call
1029 08:52:25.485802 (+3103352us) ts_tablet_manager.cc:1151] Deleting tablet 9499d0099d1c4edc843fb8d29ca3b1b6
1029 08:52:25.485811 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.485889 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3103164}
W20251029 08:52:25.475335 996 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.486626 996 leader_election.cc:304] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:25.486985 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9499d0099d1c4edc843fb8d29ca3b1b6 already in progress: creating tablet
I20251029 08:52:25.487315 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 932) took 3065 ms. Trace:
I20251029 08:52:25.487388 1176 rpcz_store.cc:276] 1029 08:52:22.422124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.423203 (+ 1079us) service_pool.cc:225] Handling call
1029 08:52:25.487216 (+3064013us) ts_tablet_manager.cc:1151] Deleting tablet 36bea657aa9044f88e50bf4e749ff01e
1029 08:52:25.487225 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.487300 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063841}
I20251029 08:52:25.487509 2485 raft_consensus.cc:2749] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:25.487773 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 36bea657aa9044f88e50bf4e749ff01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36bea657aa9044f88e50bf4e749ff01e already in progress: creating tablet
I20251029 08:52:25.488495 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 935) took 2981 ms. Trace:
I20251029 08:52:25.488582 1176 rpcz_store.cc:276] 1029 08:52:22.506701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.506782 (+ 81us) service_pool.cc:225] Handling call
1029 08:52:25.488376 (+2981594us) ts_tablet_manager.cc:1151] Deleting tablet 28d62d1e354d41f49b9ae30dbb6efc0f
1029 08:52:25.488386 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.488480 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2981363}
I20251029 08:52:25.488860 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 934) took 2989 ms. Trace:
I20251029 08:52:25.488966 753 tablet_service.cc:1505] Processing CreateTablet for tablet 3022a74550e545f49bcff1437a5cf9a5 (DEFAULT_TABLE table=default.Table_de611c4a46854ca195eb3cd4001470bf [id=89f6c5d6462e42fe8eb67ff8fea28054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.489408 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3022a74550e545f49bcff1437a5cf9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.488936 1176 rpcz_store.cc:276] 1029 08:52:22.499594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.507357 (+ 7763us) service_pool.cc:225] Handling call
1029 08:52:25.488779 (+2981422us) ts_tablet_manager.cc:1151] Deleting tablet 42de32d87ebb451090e0310e36e64277
1029 08:52:25.488786 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.488848 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2981268}
I20251029 08:52:25.490342 1176 tablet_replica.cc:333] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.490566 1176 raft_consensus.cc:2243] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.490756 1176 raft_consensus.cc:2272] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.491273 1176 ts_tablet_manager.cc:1916] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.491675 2548 raft_consensus.cc:493] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.491799 2548 raft_consensus.cc:515] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.492326 2548 leader_election.cc:290] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:25.485939 1178 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.495280 1178 ts_tablet_manager.cc:1403] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.165s user 0.002s sys 0.001s
I20251029 08:52:25.495422 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c4d793f10147fba5e4f44474575bf0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.495458 861 leader_election.cc:343] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:25.495659 1067 raft_consensus.cc:2468] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:25.495952 1178 raft_consensus.cc:359] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.496068 861 leader_election.cc:304] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.496153 1178 raft_consensus.cc:385] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.496201 1178 raft_consensus.cc:740] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.496331 2548 raft_consensus.cc:2804] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.496392 2548 raft_consensus.cc:493] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.496455 2548 raft_consensus.cc:3060] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:25.496939 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 28d62d1e354d41f49b9ae30dbb6efc0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28d62d1e354d41f49b9ae30dbb6efc0f already in progress: creating tablet
W20251029 08:52:25.497398 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 42de32d87ebb451090e0310e36e64277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42de32d87ebb451090e0310e36e64277 already in progress: creating tablet
I20251029 08:52:25.499066 2548 raft_consensus.cc:515] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.499648 2548 leader_election.cc:290] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:25.500439 861 leader_election.cc:343] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:25.500814 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8c4d793f10147fba5e4f44474575bf0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.500967 1067 raft_consensus.cc:3060] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.504393 1067 raft_consensus.cc:2468] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.504865 861 leader_election.cc:304] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.507692 2548 raft_consensus.cc:2804] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.507895 2548 raft_consensus.cc:697] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:25.508158 2548 consensus_queue.cc:237] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.510776 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 1254034e580340d7a45ba444c5bbf0b7 (DEFAULT_TABLE table=default.Table_b785a13a53ea43f6a259db613714c2c3 [id=7bd40e52c84b4256bcdcb8287e5961e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.511523 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1254034e580340d7a45ba444c5bbf0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.512673 1176 ts_tablet_manager.cc:1929] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.512835 1176 log.cc:1199] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/6d6d720fb6b54b1e8bb7b3956c48b3f9
I20251029 08:52:25.513166 1176 ts_tablet_manager.cc:1950] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.513772 1974 ts_tablet_manager.cc:1929] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.513962 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 937) took 2981 ms. Trace:
I20251029 08:52:25.514119 1176 rpcz_store.cc:276] 1029 08:52:22.532216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.532345 (+ 129us) service_pool.cc:225] Handling call
1029 08:52:25.490304 (+2957959us) ts_tablet_manager.cc:1151] Deleting tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9
1029 08:52:25.490312 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.494431 (+ 4119us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.512590 (+ 18159us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.513941 (+ 1351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2957744}
I20251029 08:52:25.514598 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432]) successfully deleted
I20251029 08:52:25.514974 1176 tablet_replica.cc:333] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.515193 1176 raft_consensus.cc:2243] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.515376 1176 raft_consensus.cc:2272] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.515902 1176 ts_tablet_manager.cc:1916] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.516391 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 899) took 3291 ms. Trace:
I20251029 08:52:25.519299 1179 rpcz_store.cc:276] 1029 08:52:22.224567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.224922 (+ 355us) service_pool.cc:225] Handling call
1029 08:52:25.202158 (+2977236us) ts_tablet_manager.cc:1151] Deleting tablet c66cbb495f5142d79a674f7d5adc505e
1029 08:52:25.202167 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.310885 (+108718us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.379398 (+ 68513us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.516372 (+136974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977036}
I20251029 08:52:25.516476 1183 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.519049 1178 consensus_queue.cc:260] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.522390 1183 ts_tablet_manager.cc:1403] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20251029 08:52:25.522771 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c66cbb495f5142d79a674f7d5adc505e (table default.Table_433d1398ff1f4ba88a75224fe2e4d069 [id=c9a2a27d7cb04c4badbd4148bcb80b68]) successfully deleted
I20251029 08:52:25.523125 1183 raft_consensus.cc:359] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.523324 1183 raft_consensus.cc:385] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.523387 1183 raft_consensus.cc:740] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.523581 1183 consensus_queue.cc:260] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.524047 1183 ts_tablet_manager.cc:1434] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:25.524844 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3e9067e3526a433e8b4d64a6a5b739a5" } modify_external_catalogs: true
I20251029 08:52:25.525041 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3e9067e3526a433e8b4d64a6a5b739a5" } modify_external_catalogs: true
I20251029 08:52:25.527101 1178 ts_tablet_manager.cc:1434] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20251029 08:52:25.527415 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 900) took 3289 ms. Trace:
I20251029 08:52:25.527524 1183 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.527748 1179 rpcz_store.cc:276] 1029 08:52:22.238240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.238292 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:25.523264 (+3284972us) ts_tablet_manager.cc:1151] Deleting tablet b582dfeab662456ba4730f5c80455d83
1029 08:52:25.523274 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.527386 (+ 4112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3284786}
I20251029 08:52:25.528316 1178 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:25.528816 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b582dfeab662456ba4730f5c80455d83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b582dfeab662456ba4730f5c80455d83 already in progress: creating tablet
I20251029 08:52:25.529239 1183 tablet_bootstrap.cc:654] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.532634 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 901) took 3260 ms. Trace:
I20251029 08:52:25.534528 1179 rpcz_store.cc:276] 1029 08:52:22.272372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.272438 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:25.532486 (+3260048us) ts_tablet_manager.cc:1151] Deleting tablet 0b61c710897f43f798d88d92c4676d0d
1029 08:52:25.532499 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.532613 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259844}
I20251029 08:52:25.538254 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_df009bebc8ca4185b078a085e10c3499" } modify_external_catalogs: true
I20251029 08:52:25.538447 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_df009bebc8ca4185b078a085e10c3499" } modify_external_catalogs: true
I20251029 08:52:25.538786 1176 ts_tablet_manager.cc:1929] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.538887 1176 log.cc:1199] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/930314c8f90f462990f93fab8f7790c2
I20251029 08:52:25.539158 1176 ts_tablet_manager.cc:1950] T 930314c8f90f462990f93fab8f7790c2 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.539892 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 940) took 2491 ms. Trace:
I20251029 08:52:25.539963 1176 rpcz_store.cc:276] 1029 08:52:23.048264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.048329 (+ 65us) service_pool.cc:225] Handling call
1029 08:52:25.514924 (+2466595us) ts_tablet_manager.cc:1151] Deleting tablet 930314c8f90f462990f93fab8f7790c2
1029 08:52:25.514933 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.522896 (+ 7963us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.538693 (+ 15797us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.539872 (+ 1179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466349}
I20251029 08:52:25.542052 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 941) took 2493 ms. Trace:
I20251029 08:52:25.544278 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 930314c8f90f462990f93fab8f7790c2 (table default.Table_11e47ff2a71e439ebfb0e9ba79a95267 [id=5691028224ad4d21b0c0ea667274b8af]) successfully deleted
I20251029 08:52:25.545032 753 tablet_service.cc:1505] Processing CreateTablet for tablet 1254034e580340d7a45ba444c5bbf0b7 (DEFAULT_TABLE table=default.Table_b785a13a53ea43f6a259db613714c2c3 [id=7bd40e52c84b4256bcdcb8287e5961e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.546923 1176 rpcz_store.cc:276] 1029 08:52:23.048383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.048612 (+ 229us) service_pool.cc:225] Handling call
1029 08:52:25.541841 (+2493229us) ts_tablet_manager.cc:1151] Deleting tablet dbad8c4014b6423cb20e3e5c22ed69f1
1029 08:52:25.541859 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.542018 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2493059}
I20251029 08:52:25.547201 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1254034e580340d7a45ba444c5bbf0b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:25.547458 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet dbad8c4014b6423cb20e3e5c22ed69f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbad8c4014b6423cb20e3e5c22ed69f1 already in progress: creating tablet
I20251029 08:52:25.547840 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 947) took 2498 ms. Trace:
I20251029 08:52:25.547951 1176 rpcz_store.cc:276] 1029 08:52:23.049669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.049714 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:25.547734 (+2498020us) ts_tablet_manager.cc:1151] Deleting tablet 5c93076ca44542a5ae19e2c73dba95cb
1029 08:52:25.547743 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.547826 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463480}
W20251029 08:52:25.548452 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5c93076ca44542a5ae19e2c73dba95cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c93076ca44542a5ae19e2c73dba95cb already in progress: creating tablet
I20251029 08:52:25.549655 1176 tablet_replica.cc:333] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.549906 1176 raft_consensus.cc:2243] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.550091 1176 raft_consensus.cc:2272] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.550691 1176 ts_tablet_manager.cc:1916] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.557106 1176 ts_tablet_manager.cc:1929] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.563710 1176 log.cc:1199] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/24834985a71349b9865840e9b5537d72
I20251029 08:52:25.564119 1176 ts_tablet_manager.cc:1950] T 24834985a71349b9865840e9b5537d72 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.564921 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 949) took 2515 ms. Trace:
I20251029 08:52:25.565061 1176 rpcz_store.cc:276] 1029 08:52:23.049928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.049981 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:25.549615 (+2499634us) ts_tablet_manager.cc:1151] Deleting tablet 24834985a71349b9865840e9b5537d72
1029 08:52:25.549624 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.554065 (+ 4441us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.557019 (+ 2954us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.564902 (+ 7883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465305}
I20251029 08:52:25.565574 1176 tablet_replica.cc:333] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.565522 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 24834985a71349b9865840e9b5537d72 (table default.Table_0b00cd7c457d4204965de864c88be5e6 [id=0158571f3f9246aeafbfd8b8b1548482]) successfully deleted
I20251029 08:52:25.565827 1176 raft_consensus.cc:2243] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.566107 1176 raft_consensus.cc:2272] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:25.566699 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0b61c710897f43f798d88d92c4676d0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b61c710897f43f798d88d92c4676d0d already in progress: creating tablet
I20251029 08:52:25.567145 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 904) took 3134 ms. Trace:
I20251029 08:52:25.567437 1176 ts_tablet_manager.cc:1916] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.567325 1179 rpcz_store.cc:276] 1029 08:52:22.433148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.433557 (+ 409us) service_pool.cc:225] Handling call
1029 08:52:25.567012 (+3133455us) ts_tablet_manager.cc:1151] Deleting tablet 9499d0099d1c4edc843fb8d29ca3b1b6
1029 08:52:25.567022 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.567125 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3041703}
W20251029 08:52:25.567919 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9499d0099d1c4edc843fb8d29ca3b1b6 already in progress: creating tablet
I20251029 08:52:25.568440 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 905) took 3135 ms. Trace:
I20251029 08:52:25.568746 1179 rpcz_store.cc:276] 1029 08:52:22.433237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.435966 (+ 2729us) service_pool.cc:225] Handling call
1029 08:52:25.568275 (+3132309us) ts_tablet_manager.cc:1151] Deleting tablet d09ed524e97c4c1bb3c4aaea876415ea
1029 08:52:25.568288 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.568417 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027738}
W20251029 08:52:25.569330 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d09ed524e97c4c1bb3c4aaea876415ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d09ed524e97c4c1bb3c4aaea876415ea already in progress: creating tablet
I20251029 08:52:25.569726 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 908) took 3136 ms. Trace:
I20251029 08:52:25.569832 1179 rpcz_store.cc:276] 1029 08:52:22.433529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.433907 (+ 378us) service_pool.cc:225] Handling call
1029 08:52:25.569630 (+3135723us) ts_tablet_manager.cc:1151] Deleting tablet 36bea657aa9044f88e50bf4e749ff01e
1029 08:52:25.569639 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.569713 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028654}
W20251029 08:52:25.570237 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 36bea657aa9044f88e50bf4e749ff01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36bea657aa9044f88e50bf4e749ff01e already in progress: creating tablet
I20251029 08:52:25.570585 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 911) took 3044 ms. Trace:
I20251029 08:52:25.570686 1179 rpcz_store.cc:276] 1029 08:52:22.525710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.526193 (+ 483us) service_pool.cc:225] Handling call
1029 08:52:25.570494 (+3044301us) ts_tablet_manager.cc:1151] Deleting tablet 42de32d87ebb451090e0310e36e64277
1029 08:52:25.570502 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.570571 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3029305}
W20251029 08:52:25.571098 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 42de32d87ebb451090e0310e36e64277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42de32d87ebb451090e0310e36e64277 already in progress: creating tablet
I20251029 08:52:25.571487 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 912) took 3045 ms. Trace:
I20251029 08:52:25.571591 1179 rpcz_store.cc:276] 1029 08:52:22.525793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.526252 (+ 459us) service_pool.cc:225] Handling call
1029 08:52:25.571384 (+3045132us) ts_tablet_manager.cc:1151] Deleting tablet 28d62d1e354d41f49b9ae30dbb6efc0f
1029 08:52:25.571393 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.571472 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3029673}
W20251029 08:52:25.571977 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 28d62d1e354d41f49b9ae30dbb6efc0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28d62d1e354d41f49b9ae30dbb6efc0f already in progress: creating tablet
I20251029 08:52:25.572383 1179 tablet_replica.cc:333] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:25.572695 1179 raft_consensus.cc:2243] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.572947 1179 raft_consensus.cc:2272] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.573697 1179 ts_tablet_manager.cc:1916] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.574831 1178 tablet_bootstrap.cc:654] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.578341 1974 log.cc:1199] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:25.579092 1974 ts_tablet_manager.cc:1950] T a2b7853b85d24a04b8b11caaa4fe8c4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.579463 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 961) took 2774 ms. Trace:
I20251029 08:52:25.579599 1974 rpcz_store.cc:276] 1029 08:52:22.804531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.804581 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:25.317258 (+2512677us) ts_tablet_manager.cc:1151] Deleting tablet a2b7853b85d24a04b8b11caaa4fe8c4d
1029 08:52:25.317266 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.510694 (+193428us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.513688 (+ 2994us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.579444 (+ 65756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2512485}
I20251029 08:52:25.580114 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a2b7853b85d24a04b8b11caaa4fe8c4d (table default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]) successfully deleted
I20251029 08:52:25.528086 888 tablet_service.cc:1505] Processing CreateTablet for tablet 1254034e580340d7a45ba444c5bbf0b7 (DEFAULT_TABLE table=default.Table_b785a13a53ea43f6a259db613714c2c3 [id=7bd40e52c84b4256bcdcb8287e5961e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.546816 2451 raft_consensus.cc:493] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.580525 2451 raft_consensus.cc:515] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.581100 2451 leader_election.cc:290] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:25.582373 861 leader_election.cc:343] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:25.582707 892 tablet_service.cc:1505] Processing CreateTablet for tablet 7838885086ef4ee98bcca8fd3dfe5fcc (DEFAULT_TABLE table=default.Table_65ecb08cc196408da5b8ac8514415da9 [id=b37f1c4c05494843bcf127511b595d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.583073 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7838885086ef4ee98bcca8fd3dfe5fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.583659 758 tablet_service.cc:1505] Processing CreateTablet for tablet 7838885086ef4ee98bcca8fd3dfe5fcc (DEFAULT_TABLE table=default.Table_65ecb08cc196408da5b8ac8514415da9 [id=b37f1c4c05494843bcf127511b595d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.584069 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7838885086ef4ee98bcca8fd3dfe5fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.584136 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1254034e580340d7a45ba444c5bbf0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.584991 889 tablet_service.cc:1505] Processing CreateTablet for tablet 33c7e63019634c229d824b2287683151 (DEFAULT_TABLE table=default.Table_af732cb4b90d49819386fe8dbb50ce73 [id=a0c554630ed843f894b51d8cc084cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.585310 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7e63019634c229d824b2287683151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.586030 1176 ts_tablet_manager.cc:1929] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.586113 1176 log.cc:1199] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e2b87d133f8a4bd1877e07a1a8758147
I20251029 08:52:25.586392 1176 ts_tablet_manager.cc:1950] T e2b87d133f8a4bd1877e07a1a8758147 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.586939 1974 tablet_replica.cc:333] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.587064 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 950) took 2537 ms. Trace:
I20251029 08:52:25.587136 1974 raft_consensus.cc:2243] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.587137 1176 rpcz_store.cc:276] 1029 08:52:23.050019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.050192 (+ 173us) service_pool.cc:225] Handling call
1029 08:52:25.565528 (+2515336us) ts_tablet_manager.cc:1151] Deleting tablet e2b87d133f8a4bd1877e07a1a8758147
1029 08:52:25.565537 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.570658 (+ 5121us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.585942 (+ 15284us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.587047 (+ 1105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2480479}
I20251029 08:52:25.587281 1974 raft_consensus.cc:2272] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.587543 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e2b87d133f8a4bd1877e07a1a8758147 (table default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da [id=85dda4d8658f4607bafddcfba7b8678a]) successfully deleted
I20251029 08:52:25.587880 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 946) took 2538 ms. Trace:
I20251029 08:52:25.587947 1176 rpcz_store.cc:276] 1029 08:52:23.049580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.050137 (+ 557us) service_pool.cc:225] Handling call
1029 08:52:25.587771 (+2537634us) ts_tablet_manager.cc:1151] Deleting tablet a186df0db24c4d22abee7ec57bcb199b
1029 08:52:25.587781 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.587866 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502707}
W20251029 08:52:25.588985 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a186df0db24c4d22abee7ec57bcb199b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a186df0db24c4d22abee7ec57bcb199b already in progress: creating tablet
I20251029 08:52:25.589344 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 952) took 2498 ms. Trace:
I20251029 08:52:25.589409 1974 ts_tablet_manager.cc:1916] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.589463 1176 rpcz_store.cc:276] 1029 08:52:23.090600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.090660 (+ 60us) service_pool.cc:225] Handling call
1029 08:52:25.589248 (+2498588us) ts_tablet_manager.cc:1151] Deleting tablet 61b901ffd7ae47f487a355f08c076a22
1029 08:52:25.589257 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.589329 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498375}
W20251029 08:52:25.589901 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 61b901ffd7ae47f487a355f08c076a22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b901ffd7ae47f487a355f08c076a22 already in progress: creating tablet
I20251029 08:52:25.590256 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 953) took 2499 ms. Trace:
I20251029 08:52:25.590369 1176 rpcz_store.cc:276] 1029 08:52:23.091020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.091062 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:25.590165 (+2499103us) ts_tablet_manager.cc:1151] Deleting tablet 6691e1944ae947b49e4431f31ba9ae41
1029 08:52:25.590173 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.590241 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498889}
W20251029 08:52:25.590796 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6691e1944ae947b49e4431f31ba9ae41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6691e1944ae947b49e4431f31ba9ae41 already in progress: creating tablet
I20251029 08:52:25.591130 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 954) took 2494 ms. Trace:
I20251029 08:52:25.591230 1176 rpcz_store.cc:276] 1029 08:52:23.096219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.096275 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:25.591039 (+2494764us) ts_tablet_manager.cc:1151] Deleting tablet c18e0e01fdc54b8dab97ec339d2aaa07
1029 08:52:25.591048 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.591116 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494478}
W20251029 08:52:25.591686 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c18e0e01fdc54b8dab97ec339d2aaa07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c18e0e01fdc54b8dab97ec339d2aaa07 already in progress: creating tablet
I20251029 08:52:25.576509 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 7838885086ef4ee98bcca8fd3dfe5fcc (DEFAULT_TABLE table=default.Table_65ecb08cc196408da5b8ac8514415da9 [id=b37f1c4c05494843bcf127511b595d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.592103 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 956) took 2469 ms. Trace:
I20251029 08:52:25.592219 1176 rpcz_store.cc:276] 1029 08:52:23.122912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.123365 (+ 453us) service_pool.cc:225] Handling call
1029 08:52:25.591939 (+2468574us) ts_tablet_manager.cc:1151] Deleting tablet 62597189ecb047788b83788b514e9099
1029 08:52:25.591948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.592041 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467460}
I20251029 08:52:25.592327 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7838885086ef4ee98bcca8fd3dfe5fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.592422 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 33c7e63019634c229d824b2287683151 (DEFAULT_TABLE table=default.Table_af732cb4b90d49819386fe8dbb50ce73 [id=a0c554630ed843f894b51d8cc084cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.592816 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7e63019634c229d824b2287683151. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:25.593093 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 62597189ecb047788b83788b514e9099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62597189ecb047788b83788b514e9099 already in progress: creating tablet
I20251029 08:52:25.593444 1176 tablet_replica.cc:333] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.593678 1176 raft_consensus.cc:2243] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.593869 1176 raft_consensus.cc:2272] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.594413 1176 ts_tablet_manager.cc:1916] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.577579 1183 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.596000 1183 ts_tablet_manager.cc:1403] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.007s
I20251029 08:52:25.581775 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627bc8ff99394fa2abcca15d62e5ea10" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.583177 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 3022a74550e545f49bcff1437a5cf9a5 (DEFAULT_TABLE table=default.Table_de611c4a46854ca195eb3cd4001470bf [id=89f6c5d6462e42fe8eb67ff8fea28054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.596604 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3022a74550e545f49bcff1437a5cf9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.596760 1183 raft_consensus.cc:359] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.596921 1183 raft_consensus.cc:385] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.596982 1183 raft_consensus.cc:740] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.597175 1183 consensus_queue.cc:260] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.597676 1183 ts_tablet_manager.cc:1434] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:25.597945 1183 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.602708 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0988621e83c64bec83ee43c523b09b20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.603255 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0988621e83c64bec83ee43c523b09b20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.607291 2366 raft_consensus.cc:493] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.607409 2366 raft_consensus.cc:515] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.559710 2611 tablet.cc:1722] T 69d27944f3c14a6eb8331a904d6faa40 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:25.596257 1065 raft_consensus.cc:2468] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:25.607960 2366 leader_election.cc:290] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.608705 1176 ts_tablet_manager.cc:1929] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.608807 1176 log.cc:1199] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:25.609109 1176 ts_tablet_manager.cc:1950] T 9429f6abb50c4f66b2e1b565f303962d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.610085 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:25.610270 783 raft_consensus.cc:2468] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:25.610785 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 971) took 2154 ms. Trace:
I20251029 08:52:25.610924 1183 tablet_bootstrap.cc:654] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.610982 998 leader_election.cc:304] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:25.611260 2366 raft_consensus.cc:2804] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.611326 2366 raft_consensus.cc:493] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.611379 2366 raft_consensus.cc:3060] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.612104 1176 rpcz_store.cc:276] 1029 08:52:23.456301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.456592 (+ 291us) service_pool.cc:225] Handling call
1029 08:52:25.593361 (+2136769us) ts_tablet_manager.cc:1151] Deleting tablet 9429f6abb50c4f66b2e1b565f303962d
1029 08:52:25.593397 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.599173 (+ 5776us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.608622 (+ 9449us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.610768 (+ 2146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136562}
I20251029 08:52:25.612597 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9429f6abb50c4f66b2e1b565f303962d (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:25.612954 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 972) took 1908 ms. Trace:
I20251029 08:52:25.613018 1176 rpcz_store.cc:276] 1029 08:52:23.704096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.704576 (+ 480us) service_pool.cc:225] Handling call
1029 08:52:25.612832 (+1908256us) ts_tablet_manager.cc:1151] Deleting tablet 9493653c2e354458923e9e8dd0da0c60
1029 08:52:25.612841 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.612939 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907988}
W20251029 08:52:25.613400 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9493653c2e354458923e9e8dd0da0c60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9493653c2e354458923e9e8dd0da0c60 already in progress: creating tablet
I20251029 08:52:25.613924 2366 raft_consensus.cc:515] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.614977 2366 leader_election.cc:290] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.615496 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:25.615675 783 raft_consensus.cc:3060] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.574478 1177 tablet_bootstrap.cc:654] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.594952 2451 consensus_queue.cc:1048] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.608798 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.617877 1178 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.618001 1178 ts_tablet_manager.cc:1403] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.000s
W20251029 08:52:25.618768 996 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.618777 1178 raft_consensus.cc:359] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.618916 1178 raft_consensus.cc:385] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.618976 1178 raft_consensus.cc:740] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.619221 1178 consensus_queue.cc:260] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.619499 783 raft_consensus.cc:2468] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.610770 861 leader_election.cc:304] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.620355 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 974) took 1727 ms. Trace:
I20251029 08:52:25.620415 998 leader_election.cc:304] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:25.620450 1176 rpcz_store.cc:276] 1029 08:52:23.892492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.892569 (+ 77us) service_pool.cc:225] Handling call
1029 08:52:25.620107 (+1727538us) ts_tablet_manager.cc:1151] Deleting tablet a92cc4a93a584dffaafe21316fe8dc06
1029 08:52:25.620232 (+ 125us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.620337 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727319}
I20251029 08:52:25.620668 2451 raft_consensus.cc:2804] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.620745 2451 raft_consensus.cc:493] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.620791 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 973) took 1728 ms. Trace:
I20251029 08:52:25.620811 2451 raft_consensus.cc:3060] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.620854 1176 rpcz_store.cc:276] 1029 08:52:23.892360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.905584 (+ 13224us) service_pool.cc:225] Handling call
1029 08:52:25.620662 (+1715078us) ts_tablet_manager.cc:1151] Deleting tablet d5bc2f2f5d6e418999fa40a6fadbbc71
1029 08:52:25.620671 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.620776 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714698}
I20251029 08:52:25.621140 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 975) took 1707 ms. Trace:
I20251029 08:52:25.621198 1176 rpcz_store.cc:276] 1029 08:52:23.913565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.913623 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:25.621050 (+1707427us) ts_tablet_manager.cc:1151] Deleting tablet fb22af7515084e099b41f9c1c92c067a
1029 08:52:25.621057 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.621127 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1707218}
I20251029 08:52:25.621398 1176 tablet_replica.cc:333] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.621699 1178 ts_tablet_manager.cc:1434] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 08:52:25.621999 1178 tablet_bootstrap.cc:492] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:25.622006 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a92cc4a93a584dffaafe21316fe8dc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92cc4a93a584dffaafe21316fe8dc06 already in progress: creating tablet
W20251029 08:52:25.622225 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bc2f2f5d6e418999fa40a6fadbbc71 already in progress: creating tablet
W20251029 08:52:25.622429 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet fb22af7515084e099b41f9c1c92c067a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb22af7515084e099b41f9c1c92c067a already in progress: creating tablet
I20251029 08:52:25.623466 2451 raft_consensus.cc:515] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.624369 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627bc8ff99394fa2abcca15d62e5ea10" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.624521 1067 raft_consensus.cc:3060] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.624995 1176 raft_consensus.cc:2243] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.625217 1176 raft_consensus.cc:2272] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.625732 1176 ts_tablet_manager.cc:1916] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.627130 1067 raft_consensus.cc:2468] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.559252 2554 raft_consensus.cc:515] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.627552 861 leader_election.cc:304] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:25.627861 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6689cf0804f946c18ebc555a191549d4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:25.628335 996 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.629912 2366 raft_consensus.cc:2804] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.630096 2366 raft_consensus.cc:697] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.630319 2366 consensus_queue.cc:237] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.632994 2295 raft_consensus.cc:2804] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.633114 2295 raft_consensus.cc:697] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:25.633329 2295 consensus_queue.cc:237] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.635190 1176 ts_tablet_manager.cc:1929] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:25.636475 2451 leader_election.cc:290] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:25.637185 861 leader_election.cc:343] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:25.640841 1176 log.cc:1199] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e687f593eead488db4bb445a63beeec7
I20251029 08:52:25.641264 1176 ts_tablet_manager.cc:1950] T e687f593eead488db4bb445a63beeec7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.642046 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 976) took 1421 ms. Trace:
I20251029 08:52:25.642176 1176 rpcz_store.cc:276] 1029 08:52:24.220209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.220796 (+ 587us) service_pool.cc:225] Handling call
1029 08:52:25.621364 (+1400568us) ts_tablet_manager.cc:1151] Deleting tablet e687f593eead488db4bb445a63beeec7
1029 08:52:25.621371 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.631672 (+ 10301us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.635110 (+ 3438us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.642030 (+ 6920us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3390,"tablet-delete.queue_time_us":1400381}
I20251029 08:52:25.642757 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e687f593eead488db4bb445a63beeec7 (table default.Table_d707b992dd69414897e525d86fb4b002 [id=a23a597c8fd343769b517507845104ff]) successfully deleted
I20251029 08:52:25.643076 1176 tablet_replica.cc:333] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.643292 1176 raft_consensus.cc:2243] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.643527 1176 raft_consensus.cc:2272] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.646848 1176 ts_tablet_manager.cc:1916] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.651902 592 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_2b24b34725b34a2189255d1a59532f6b [id=b253b07dae944d078fe7589af7e3adfd] alter complete (version 1)
I20251029 08:52:25.656283 2458 raft_consensus.cc:493] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.656488 2458 raft_consensus.cc:515] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.657089 2458 leader_election.cc:290] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:25.657856 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.658051 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.658468 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af913080bfe44a86a76704ffa32cd9e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.659101 718 leader_election.cc:343] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.659677 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af913080bfe44a86a76704ffa32cd9e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.660032 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:25.660408 720 leader_election.cc:343] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.660516 720 leader_election.cc:304] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.660955 2458 raft_consensus.cc:2749] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.661083 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.661847 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.662739 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.662665 748 tablet_service.cc:1505] Processing CreateTablet for tablet 33c7e63019634c229d824b2287683151 (DEFAULT_TABLE table=default.Table_af732cb4b90d49819386fe8dbb50ce73 [id=a0c554630ed843f894b51d8cc084cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.663141 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d8b2eaecf62d466a9bfe3a63d1b9d838"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:25.663409 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:25.663739 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8b2eaecf62d466a9bfe3a63d1b9d838 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.663975 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.664590 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.665166 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4cf1783778c64509b2a98c717f0477a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:25.665269 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:25.665668 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cf1783778c64509b2a98c717f0477a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.665913 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.667788 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.668212 1067 raft_consensus.cc:3060] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.668259 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7e63019634c229d824b2287683151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.668870 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a2b7853b85d24a04b8b11caaa4fe8c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.671509 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 627bc8ff99394fa2abcca15d62e5ea10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.671808 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c4d57a59684723bfd27c7dda302ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.672115 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 652903cbbdb042d780e7d71fc2c0ab4d with delete_type TABLET_DATA_DELETED (replaced by tablet 07bf5d693be240f18bacccc81a6ea5d4) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.672479 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e0e85daedef44a1af1e4fc46ceeab65) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.672783 892 tablet_service.cc:1552] Processing DeleteTablet for tablet aafa846cb2144cec9ce4b28e61d91961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.673036 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 43c3074caee74069a8a7f1b87721d726 with delete_type TABLET_DATA_DELETED (replaced by tablet 527a83465a844849811b0f9d26428af0) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.673478 892 tablet_service.cc:1505] Processing CreateTablet for tablet c24db730f7f342fead8768e3dba28372 (DEFAULT_TABLE table=default.Table_6c752dc40c034ebaa141deb809297b0b [id=cc3fc31b5c8146ab8dc6c17f0697be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.673858 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24db730f7f342fead8768e3dba28372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.674230 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet b0ff0658e0274cf49c389ad602d91b03 (table default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8]). Expected version 3 got 0
I20251029 08:52:25.674448 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:25.674810 596 catalog_manager.cc:5649] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:25.675384 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet af913080bfe44a86a76704ffa32cd9e1 (table default.Table_7fd53f3b59d14ad39715bc62e513ab79 [id=88bde60591764b37b719bdeea15ef0cd]). Expected version 1 got 0
I20251029 08:52:25.675652 1067 raft_consensus.cc:2468] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.676036 861 leader_election.cc:304] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:25.676306 2451 raft_consensus.cc:2804] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.676445 2451 raft_consensus.cc:697] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:25.676668 2451 consensus_queue.cc:237] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.678829 2395 log.cc:927] Time spent T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Append to log took a long time: real 0.123s user 0.002s sys 0.000s
I20251029 08:52:25.688174 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7091ff6bed4e3a8447d9de8ea9fb48" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:25.688349 783 raft_consensus.cc:3060] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.691236 1035 tablet_service.cc:1505] Processing CreateTablet for tablet c24db730f7f342fead8768e3dba28372 (DEFAULT_TABLE table=default.Table_6c752dc40c034ebaa141deb809297b0b [id=cc3fc31b5c8146ab8dc6c17f0697be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.691638 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24db730f7f342fead8768e3dba28372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.692359 783 raft_consensus.cc:2468] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.697806 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 71ad7afe989143289da63d105c146900 (DEFAULT_TABLE table=default.Table_755a398e8bef4cbd9268a01dff6fd627 [id=4252463ab98e405b8a600b54a8fbb424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.698150 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ad7afe989143289da63d105c146900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.700451 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7eeac248fb405096ea06ae80aa884c (DEFAULT_TABLE table=default.Table_d15768ec762b4df6ad8b993fad1087a1 [id=9a66a7fc745e4c3e8cf13063c45ca4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.700783 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7eeac248fb405096ea06ae80aa884c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.700981 1183 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.701099 1183 ts_tablet_manager.cc:1403] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.014s
I20251029 08:52:25.701844 1183 raft_consensus.cc:359] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.702004 1183 raft_consensus.cc:385] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.702057 1183 raft_consensus.cc:740] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.702240 1183 consensus_queue.cc:260] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.702762 1183 ts_tablet_manager.cc:1434] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:25.703016 1183 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.704516 1183 tablet_bootstrap.cc:654] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.704803 1177 tablet_bootstrap.cc:492] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.704928 1177 ts_tablet_manager.cc:1403] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.221s user 0.005s sys 0.001s
I20251029 08:52:25.705600 1177 raft_consensus.cc:359] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.705746 1177 raft_consensus.cc:385] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.705788 1177 raft_consensus.cc:740] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.705965 1177 consensus_queue.cc:260] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.707486 748 tablet_service.cc:1505] Processing CreateTablet for tablet 71ad7afe989143289da63d105c146900 (DEFAULT_TABLE table=default.Table_755a398e8bef4cbd9268a01dff6fd627 [id=4252463ab98e405b8a600b54a8fbb424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.707495 758 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7eeac248fb405096ea06ae80aa884c (DEFAULT_TABLE table=default.Table_d15768ec762b4df6ad8b993fad1087a1 [id=9a66a7fc745e4c3e8cf13063c45ca4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.707871 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ad7afe989143289da63d105c146900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.708072 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7eeac248fb405096ea06ae80aa884c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.714596 783 raft_consensus.cc:1275] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Refusing update from remote peer 730d0f5b7abf43a293b7df13d01ce4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 08:52:25.714612 996 consensus_peers.cc:597] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.714833 2554 leader_election.cc:290] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.715224 2485 consensus_queue.cc:1048] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.719601 2458 raft_consensus.cc:493] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.719745 2458 raft_consensus.cc:515] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.720367 2458 leader_election.cc:290] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.722268 2485 raft_consensus.cc:493] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.722388 2485 raft_consensus.cc:515] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.722914 2485 leader_election.cc:290] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.726181 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ab56045bd814d59831a98618c759af4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.726667 720 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:25.728446 998 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726
I20251029 08:52:25.731951 753 tablet_service.cc:1505] Processing CreateTablet for tablet c24db730f7f342fead8768e3dba28372 (DEFAULT_TABLE table=default.Table_6c752dc40c034ebaa141deb809297b0b [id=cc3fc31b5c8146ab8dc6c17f0697be5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.732429 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24db730f7f342fead8768e3dba28372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.737021 1176 ts_tablet_manager.cc:1929] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:25.739356 1176 log.cc:1199] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/10baa0547180493f872a23614cf4c608
I20251029 08:52:25.739833 1176 ts_tablet_manager.cc:1950] T 10baa0547180493f872a23614cf4c608 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.719612 892 tablet_service.cc:1505] Processing CreateTablet for tablet 71ad7afe989143289da63d105c146900 (DEFAULT_TABLE table=default.Table_755a398e8bef4cbd9268a01dff6fd627 [id=4252463ab98e405b8a600b54a8fbb424]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.740401 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ad7afe989143289da63d105c146900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.720427 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8494757576ff4fee845e4a9d48fb0d5a (DEFAULT_TABLE table=default.Table_f04c7f07ea40494aaf173ed78172c756 [id=ae52fe000d344690921861e0fdc292b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.744402 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8494757576ff4fee845e4a9d48fb0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.745900 1974 ts_tablet_manager.cc:1929] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.745987 1974 log.cc:1199] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:25.745999 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0c2072c16fce43278f1e32b9672187e7
I20251029 08:52:25.746304 1974 ts_tablet_manager.cc:1950] T bbc4345cabbc476485b6a040a076ddaf P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.746614 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 960) took 2942 ms. Trace:
I20251029 08:52:25.746680 1974 rpcz_store.cc:276] 1029 08:52:22.804400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.804859 (+ 459us) service_pool.cc:225] Handling call
1029 08:52:25.586898 (+2782039us) ts_tablet_manager.cc:1151] Deleting tablet bbc4345cabbc476485b6a040a076ddaf
1029 08:52:25.586907 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.742798 (+155891us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.745824 (+ 3026us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.746599 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2781902}
I20251029 08:52:25.721549 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ab56045bd814d59831a98618c759af4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.725668 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43c3074caee74069a8a7f1b87721d726" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.748314 924 raft_consensus.cc:2468] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:25.748481 718 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.748596 718 leader_election.cc:304] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.748979 2458 raft_consensus.cc:2749] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.749233 996 leader_election.cc:304] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.749861 2485 raft_consensus.cc:2804] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.749881 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.749949 2485 raft_consensus.cc:493] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.750006 2485 raft_consensus.cc:3060] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.750869 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 41622daeb1e0419fa88cc26c0405c2db
I20251029 08:52:25.623973 1178 tablet_bootstrap.cc:654] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.752636 2485 raft_consensus.cc:515] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.753521 2485 leader_election.cc:290] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.753891 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.754158 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5498826a9afd4fd5a6b88b0a465fff40
I20251029 08:52:25.754555 591 catalog_manager.cc:6719] ReplaceTablet: tablet 5498826a9afd4fd5a6b88b0a465fff40 deleted and replaced by tablet c5d9527d294c432db2ccf60af84731b0
I20251029 08:52:25.754515 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43c3074caee74069a8a7f1b87721d726" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:25.754671 924 raft_consensus.cc:3060] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.754715 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c14c63f74f74a18902a17a4d210a90d from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:25.756779 998 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726
I20251029 08:52:25.757218 924 raft_consensus.cc:2468] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.757823 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 41622daeb1e0419fa88cc26c0405c2db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.757931 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d9527d294c432db2ccf60af84731b0) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.757977 996 leader_election.cc:304] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:25.758296 2485 raft_consensus.cc:2804] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.758455 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 41622daeb1e0419fa88cc26c0405c2db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.758517 2485 raft_consensus.cc:697] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.758635 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d9527d294c432db2ccf60af84731b0) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.759389 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.759670 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 41622daeb1e0419fa88cc26c0405c2db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.758738 2485 consensus_queue.cc:237] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.760079 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 5498826a9afd4fd5a6b88b0a465fff40 with delete_type TABLET_DATA_DELETED (replaced by tablet c5d9527d294c432db2ccf60af84731b0) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.760113 2548 consensus_queue.cc:1048] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.760289 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bbc4345cabbc476485b6a040a076ddaf (table default.Table_64f9de5287d54b348a406a42a7c379ab [id=0e01efd35bcd43dc88e2d92f0734b958]) successfully deleted
I20251029 08:52:25.761336 2545 tablet.cc:1722] T 026eef7a3cba40d0a2c090485cea55e0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:25.730921 890 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7eeac248fb405096ea06ae80aa884c (DEFAULT_TABLE table=default.Table_d15768ec762b4df6ad8b993fad1087a1 [id=9a66a7fc745e4c3e8cf13063c45ca4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.768373 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7eeac248fb405096ea06ae80aa884c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.768540 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2fb9ab76dcfe486e89bbe9391cab56f9
I20251029 08:52:25.771024 1177 ts_tablet_manager.cc:1434] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
I20251029 08:52:25.771296 1177 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.776620 1183 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.776803 1183 ts_tablet_manager.cc:1403] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.000s
I20251029 08:52:25.777633 1183 raft_consensus.cc:359] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.777818 1183 raft_consensus.cc:385] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.777885 1183 raft_consensus.cc:740] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.778106 1183 consensus_queue.cc:260] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.778587 1183 ts_tablet_manager.cc:1434] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:25.778863 1183 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.780159 2611 tablet.cc:1722] T 026eef7a3cba40d0a2c090485cea55e0 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:25.783751 1177 tablet_bootstrap.cc:654] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.783731 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.785095 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 977) took 1460 ms. Trace:
I20251029 08:52:25.785187 1176 rpcz_store.cc:276] 1029 08:52:24.324605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.324678 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:25.643031 (+1318353us) ts_tablet_manager.cc:1151] Deleting tablet 10baa0547180493f872a23614cf4c608
1029 08:52:25.643040 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.679216 (+ 36176us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.736926 (+ 57710us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.785073 (+ 48147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318080}
I20251029 08:52:25.785516 1176 tablet_replica.cc:333] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:25.785728 1176 raft_consensus.cc:2243] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.786057 1176 raft_consensus.cc:2272] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.786628 1176 ts_tablet_manager.cc:1916] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.788892 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]) successfully deleted
I20251029 08:52:25.793550 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a3f6289022a94c0393bf7f32fb2439d4" } new_table_name: "default.Table_a050762711c3496094db8f582859d8b3" modify_external_catalogs: true
I20251029 08:52:25.797103 1176 ts_tablet_manager.cc:1929] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:25.797320 1176 log.cc:1199] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a73a891fb6034f989d29a3f837303d84
I20251029 08:52:25.797760 1176 ts_tablet_manager.cc:1950] T a73a891fb6034f989d29a3f837303d84 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:25.796954 861 consensus_peers.cc:597] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e687f593eead488db4bb445a63beeec7. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.801518 1183 tablet_bootstrap.cc:654] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.807420 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 52b697b962f849eb88741c2e2ba4643d
I20251029 08:52:25.808128 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a4078e8eb124424399751be94d3a19b0" } new_table_name: "default.Table_20da8b59e85447e7a8985e312126cc8e" modify_external_catalogs: true
I20251029 08:52:25.809438 1183 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.809463 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 58b02839973e4f018a1a6dfccf009b25
I20251029 08:52:25.809567 1183 ts_tablet_manager.cc:1403] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20251029 08:52:25.810067 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_13d40f6bbc974e0bb9e249282fe3a279"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:25.810236 1183 raft_consensus.cc:359] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.810369 861 consensus_peers.cc:597] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 652903cbbdb042d780e7d71fc2c0ab4d. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.810393 1183 raft_consensus.cc:385] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.810447 1183 raft_consensus.cc:740] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:25.810595 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13d40f6bbc974e0bb9e249282fe3a279 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.810621 1183 consensus_queue.cc:260] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.813004 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 52b697b962f849eb88741c2e2ba4643d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.813149 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 58b02839973e4f018a1a6dfccf009b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.813792 1177 tablet_bootstrap.cc:492] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.813961 1177 ts_tablet_manager.cc:1403] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.001s
I20251029 08:52:25.814033 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 58b02839973e4f018a1a6dfccf009b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.814729 1177 raft_consensus.cc:359] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.814874 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 52b697b962f849eb88741c2e2ba4643d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.814981 1177 raft_consensus.cc:385] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.815079 1177 raft_consensus.cc:740] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.815402 1177 consensus_queue.cc:260] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.816967 1179 ts_tablet_manager.cc:1929] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.817108 1179 log.cc:1199] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/930314c8f90f462990f93fab8f7790c2
I20251029 08:52:25.799993 1178 tablet_bootstrap.cc:492] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.812981 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 52b697b962f849eb88741c2e2ba4643d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.813153 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 58b02839973e4f018a1a6dfccf009b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:25.819247 1178 ts_tablet_manager.cc:1403] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.197s user 0.002s sys 0.002s
I20251029 08:52:25.820304 1178 raft_consensus.cc:359] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.820619 1178 raft_consensus.cc:385] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.820744 1178 raft_consensus.cc:740] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.821002 1178 consensus_queue.cc:260] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.821981 1178 ts_tablet_manager.cc:1434] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251029 08:52:25.822475 1178 tablet_bootstrap.cc:492] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.826014 1178 tablet_bootstrap.cc:654] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.814601 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8494757576ff4fee845e4a9d48fb0d5a (DEFAULT_TABLE table=default.Table_f04c7f07ea40494aaf173ed78172c756 [id=ae52fe000d344690921861e0fdc292b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.827255 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8494757576ff4fee845e4a9d48fb0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.833454 924 raft_consensus.cc:3060] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.836442 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 978) took 1258 ms. Trace:
I20251029 08:52:25.836544 1176 rpcz_store.cc:276] 1029 08:52:24.578153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.578216 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:25.785454 (+1207238us) ts_tablet_manager.cc:1151] Deleting tablet a73a891fb6034f989d29a3f837303d84
1029 08:52:25.785464 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.793838 (+ 8374us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.797015 (+ 3177us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.836425 (+ 39410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207018}
I20251029 08:52:25.837039 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
I20251029 08:52:25.837430 1176 tablet_replica.cc:333] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:25.837561 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 25bd82e3549442ada2503518037127fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bd82e3549442ada2503518037127fc already in progress: creating tablet
I20251029 08:52:25.837673 1176 raft_consensus.cc:2243] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.837852 1176 raft_consensus.cc:2272] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:25.837838 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5f67e7b28b4240618e1817bb47d5633d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f67e7b28b4240618e1817bb47d5633d already in progress: creating tablet
I20251029 08:52:25.838459 1176 ts_tablet_manager.cc:1916] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.840605 890 tablet_service.cc:1505] Processing CreateTablet for tablet a5076e61fc9547e2a5ef0724c3853e5b (DEFAULT_TABLE table=default.Table_2de7c30e5ad643dc9cbe84972d3f3f54 [id=bcbbb7f6285249ef9247d0a43d5011ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.840989 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5076e61fc9547e2a5ef0724c3853e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.843778 758 tablet_service.cc:1505] Processing CreateTablet for tablet 58b02839973e4f018a1a6dfccf009b25 (DEFAULT_TABLE table=default.Table_df009bebc8ca4185b078a085e10c3499 [id=31670a83e19343e09db70f6146c14885]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.844118 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b02839973e4f018a1a6dfccf009b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.848042 889 tablet_service.cc:1505] Processing CreateTablet for tablet 58b02839973e4f018a1a6dfccf009b25 (DEFAULT_TABLE table=default.Table_df009bebc8ca4185b078a085e10c3499 [id=31670a83e19343e09db70f6146c14885]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.848403 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b02839973e4f018a1a6dfccf009b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.854112 748 tablet_service.cc:1505] Processing CreateTablet for tablet a5076e61fc9547e2a5ef0724c3853e5b (DEFAULT_TABLE table=default.Table_2de7c30e5ad643dc9cbe84972d3f3f54 [id=bcbbb7f6285249ef9247d0a43d5011ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.854730 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5076e61fc9547e2a5ef0724c3853e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.855615 2458 raft_consensus.cc:493] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.855743 2458 raft_consensus.cc:515] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.839066 1178 tablet_bootstrap.cc:492] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:25.856333 1178 ts_tablet_manager.cc:1403] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.002s
I20251029 08:52:25.856392 2458 leader_election.cc:290] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.857111 1178 raft_consensus.cc:359] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.857613 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.858144 718 leader_election.cc:343] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.858405 1178 raft_consensus.cc:385] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.858551 1178 raft_consensus.cc:740] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.858815 1178 consensus_queue.cc:260] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.859396 1178 ts_tablet_manager.cc:1434] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:25.817464 1179 ts_tablet_manager.cc:1950] T 930314c8f90f462990f93fab8f7790c2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:25.818569 1183 ts_tablet_manager.cc:1434] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:25.821687 2485 raft_consensus.cc:493] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.860337 2485 raft_consensus.cc:515] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.860520 1183 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.860563 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 917) took 3303 ms. Trace:
I20251029 08:52:25.860626 1179 rpcz_store.cc:276] 1029 08:52:22.557609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.558078 (+ 469us) service_pool.cc:225] Handling call
1029 08:52:25.572325 (+3014247us) ts_tablet_manager.cc:1151] Deleting tablet 930314c8f90f462990f93fab8f7790c2
1029 08:52:25.572338 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.750720 (+178382us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.816868 (+ 66148us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.860550 (+ 43682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004157}
I20251029 08:52:25.860961 2485 leader_election.cc:290] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.860954 1179 tablet_replica.cc:333] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:25.837126 2476 consensus_queue.cc:1048] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:25.861305 1179 raft_consensus.cc:2243] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.861331 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.861481 1179 raft_consensus.cc:2272] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.861789 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 930314c8f90f462990f93fab8f7790c2 (table default.Table_11e47ff2a71e439ebfb0e9ba79a95267 [id=5691028224ad4d21b0c0ea667274b8af]) successfully deleted
I20251029 08:52:25.862048 1179 ts_tablet_manager.cc:1916] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:25.862102 996 leader_election.cc:343] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.841058 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a5076e61fc9547e2a5ef0724c3853e5b (DEFAULT_TABLE table=default.Table_2de7c30e5ad643dc9cbe84972d3f3f54 [id=bcbbb7f6285249ef9247d0a43d5011ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.862195 1183 tablet_bootstrap.cc:654] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.862473 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5076e61fc9547e2a5ef0724c3853e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.862692 1178 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:25.864156 1178 tablet_bootstrap.cc:654] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.866652 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:25.866829 783 raft_consensus.cc:2468] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:25.867762 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 963) took 2996 ms. Trace:
I20251029 08:52:25.867902 1974 rpcz_store.cc:276] 1029 08:52:22.871681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.871738 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:25.867551 (+2995813us) ts_tablet_manager.cc:1151] Deleting tablet e2b87d133f8a4bd1877e07a1a8758147
1029 08:52:25.867563 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.867740 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995636}
I20251029 08:52:25.868412 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 966) took 2935 ms. Trace:
W20251029 08:52:25.868469 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e2b87d133f8a4bd1877e07a1a8758147 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2b87d133f8a4bd1877e07a1a8758147 already in progress: creating tablet
I20251029 08:52:25.868566 1974 rpcz_store.cc:276] 1029 08:52:22.932691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.932956 (+ 265us) service_pool.cc:225] Handling call
1029 08:52:25.868281 (+2935325us) ts_tablet_manager.cc:1151] Deleting tablet 6691e1944ae947b49e4431f31ba9ae41
1029 08:52:25.868294 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.868392 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2924121}
I20251029 08:52:25.868938 998 leader_election.cc:304] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:25.869033 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6691e1944ae947b49e4431f31ba9ae41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6691e1944ae947b49e4431f31ba9ae41 already in progress: creating tablet
I20251029 08:52:25.869127 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 967) took 2919 ms. Trace:
I20251029 08:52:25.869191 2476 raft_consensus.cc:2804] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.869266 2476 raft_consensus.cc:493] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.869204 1974 rpcz_store.cc:276] 1029 08:52:22.949802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.949859 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:25.869036 (+2919177us) ts_tablet_manager.cc:1151] Deleting tablet c18e0e01fdc54b8dab97ec339d2aaa07
1029 08:52:25.869044 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.869112 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2918957}
I20251029 08:52:25.869330 2476 raft_consensus.cc:3060] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.869566 1974 tablet_replica.cc:333] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.869756 1177 ts_tablet_manager.cc:1434] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.056s user 0.001s sys 0.001s
I20251029 08:52:25.870071 1177 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.871635 1177 tablet_bootstrap.cc:654] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.871884 2476 raft_consensus.cc:515] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.872677 2476 leader_election.cc:290] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.852298 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 58b02839973e4f018a1a6dfccf009b25 (DEFAULT_TABLE table=default.Table_df009bebc8ca4185b078a085e10c3499 [id=31670a83e19343e09db70f6146c14885]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.873409 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:25.857017 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.873593 783 raft_consensus.cc:3060] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.873653 1065 raft_consensus.cc:2393] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in current term 1: Already voted for candidate 730d0f5b7abf43a293b7df13d01ce4ee in this term.
I20251029 08:52:25.874758 1974 raft_consensus.cc:2243] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:25.874881 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58b02839973e4f018a1a6dfccf009b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.874987 1974 raft_consensus.cc:2272] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.875464 1974 ts_tablet_manager.cc:1916] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.877031 720 leader_election.cc:304] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.877486 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23dc7808f8da41e3aa0af5f318cd05c1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:25.877899 783 raft_consensus.cc:2468] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:25.878005 2458 raft_consensus.cc:2749] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.838140 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.878319 998 leader_election.cc:304] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:25.878605 996 leader_election.cc:343] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.878870 2485 raft_consensus.cc:2804] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.878979 2485 raft_consensus.cc:697] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:25.879206 2485 consensus_queue.cc:237] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:25.879752 998 consensus_peers.cc:597] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:25.882107 1183 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.882236 1183 ts_tablet_manager.cc:1403] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
W20251029 08:52:25.882514 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c18e0e01fdc54b8dab97ec339d2aaa07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c18e0e01fdc54b8dab97ec339d2aaa07 already in progress: creating tablet
I20251029 08:52:25.882959 1183 raft_consensus.cc:359] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.883108 1183 raft_consensus.cc:385] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.883163 1183 raft_consensus.cc:740] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.883334 1183 consensus_queue.cc:260] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.883821 1183 ts_tablet_manager.cc:1434] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:25.884099 1183 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.885779 1183 tablet_bootstrap.cc:654] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.888473 2476 raft_consensus.cc:493] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.888640 2476 raft_consensus.cc:515] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.889467 2476 leader_election.cc:290] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:25.890008 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5ac1af8b78a4d94b0e3da14ebfffd92" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.890503 996 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:25.891274 998 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:25.891372 998 leader_election.cc:304] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.892274 2476 raft_consensus.cc:2749] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.898696 2545 tablet.cc:1722] T b0ff0658e0274cf49c389ad602d91b03 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:25.902046 1183 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.904671 2611 tablet.cc:1722] T b0ff0658e0274cf49c389ad602d91b03 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:25.909031 1179 ts_tablet_manager.cc:1929] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.909291 1177 tablet_bootstrap.cc:492] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:25.909482 1177 ts_tablet_manager.cc:1403] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251029 08:52:25.910282 1177 raft_consensus.cc:359] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.910637 1177 raft_consensus.cc:385] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.910729 1177 raft_consensus.cc:740] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.913707 1177 consensus_queue.cc:260] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.914429 1177 ts_tablet_manager.cc:1434] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:25.914530 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.914777 1177 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:25.916357 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1b99bdda9ee74a1187811dccc1daf159" } new_table_name: "default.Table_b5a6c771827043ee8999368c13c43af7" modify_external_catalogs: true
I20251029 08:52:25.917099 1177 tablet_bootstrap.cc:654] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.910563 1183 ts_tablet_manager.cc:1403] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20251029 08:52:25.921792 1179 log.cc:1199] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:25.922235 1179 ts_tablet_manager.cc:1950] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:25.922564 1183 raft_consensus.cc:359] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.922742 1183 raft_consensus.cc:385] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.922634 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 922) took 3309 ms. Trace:
I20251029 08:52:25.922793 1183 raft_consensus.cc:740] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.922878 1179 rpcz_store.cc:276] 1029 08:52:22.613458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.613529 (+ 71us) service_pool.cc:225] Handling call
1029 08:52:25.860913 (+3247384us) ts_tablet_manager.cc:1151] Deleting tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c
1029 08:52:25.860923 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.865255 (+ 4332us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.908953 (+ 43698us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.922621 (+ 13668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3244988}
I20251029 08:52:25.923063 1183 consensus_queue.cc:260] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.923333 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c097e84e85ba449185c70f18e0b3184c" } new_table_name: "default.Table_7f6426ed22804c409dd431e21ae1d0ca" modify_external_catalogs: true
I20251029 08:52:25.923446 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (table default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]) successfully deleted
I20251029 08:52:25.923553 1183 ts_tablet_manager.cc:1434] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:25.923802 1183 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.924396 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 923) took 3267 ms. Trace:
I20251029 08:52:25.911518 1974 ts_tablet_manager.cc:1929] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.924625 1974 log.cc:1199] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/fc04616301354cfe873c825a9f5d7ab6
I20251029 08:52:25.926661 1974 ts_tablet_manager.cc:1950] T fc04616301354cfe873c825a9f5d7ab6 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.924747 1179 rpcz_store.cc:276] 1029 08:52:22.656889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.656940 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:25.924276 (+3267336us) ts_tablet_manager.cc:1151] Deleting tablet a186df0db24c4d22abee7ec57bcb199b
1029 08:52:25.924285 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.924379 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267161}
W20251029 08:52:25.927202 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a186df0db24c4d22abee7ec57bcb199b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a186df0db24c4d22abee7ec57bcb199b already in progress: creating tablet
I20251029 08:52:25.927541 1179 tablet_replica.cc:333] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:25.927767 1179 raft_consensus.cc:2243] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.928172 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 970) took 2935 ms. Trace:
I20251029 08:52:25.928576 1179 raft_consensus.cc:2272] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.928714 1974 rpcz_store.cc:276] 1029 08:52:22.992675 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.992717 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:25.869527 (+2876810us) ts_tablet_manager.cc:1151] Deleting tablet fc04616301354cfe873c825a9f5d7ab6
1029 08:52:25.869535 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.890926 (+ 21391us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.911428 (+ 20502us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.928156 (+ 16728us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3959,"tablet-delete.queue_time_us":2876618}
I20251029 08:52:25.929472 1179 ts_tablet_manager.cc:1916] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.929827 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet fc04616301354cfe873c825a9f5d7ab6 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
I20251029 08:52:25.929801 758 tablet_service.cc:1505] Processing CreateTablet for tablet 777fd02e599a43f58f0e774b7717266d (DEFAULT_TABLE table=default.Table_1b99bdda9ee74a1187811dccc1daf159 [id=0baaf2b577a842e09e8fb83db4bce406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.930172 1974 tablet_replica.cc:333] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.930205 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777fd02e599a43f58f0e774b7717266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.930387 1974 raft_consensus.cc:2243] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.930752 1974 raft_consensus.cc:2272] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.932312 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0c14c63f74f74a18902a17a4d210a90d
I20251029 08:52:25.932796 591 catalog_manager.cc:6719] ReplaceTablet: tablet 0c14c63f74f74a18902a17a4d210a90d deleted and replaced by tablet 26cc2d5ccb7e49ab92c8b1da09152b27
I20251029 08:52:25.933077 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_713096b122044a3884d7d40bf3d84cb0" } new_table_name: "default.Table_7f9cb3ad1c7f439bb0b199c00294f2d4" modify_external_catalogs: true
I20251029 08:52:25.935163 593 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 713e102d9e1d40148636443a2260ccbc (table default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f]). Expected version 1 got 0
I20251029 08:52:25.922183 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 8494757576ff4fee845e4a9d48fb0d5a (DEFAULT_TABLE table=default.Table_f04c7f07ea40494aaf173ed78172c756 [id=ae52fe000d344690921861e0fdc292b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:25.936293 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97a196d34a4b48988c07034dcc971049 consensus state has no leader
I20251029 08:52:25.936316 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8494757576ff4fee845e4a9d48fb0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.936667 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_dd61164773f54438b96f450cc899ab1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.937157 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd61164773f54438b96f450cc899ab1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.937258 1183 tablet_bootstrap.cc:654] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.934263 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c14c63f74f74a18902a17a4d210a90d with delete_type TABLET_DATA_DELETED (replaced by tablet 26cc2d5ccb7e49ab92c8b1da09152b27) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:25.940627 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4fa1c998b783428e81198c6c52b5f905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.941152 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fa1c998b783428e81198c6c52b5f905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.942754 1183 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:25.942866 1183 ts_tablet_manager.cc:1403] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251029 08:52:25.943593 1183 raft_consensus.cc:359] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.943794 1183 raft_consensus.cc:385] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:25.943850 1183 raft_consensus.cc:740] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:25.943930 1974 ts_tablet_manager.cc:1916] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.944104 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c14c63f74f74a18902a17a4d210a90d with delete_type TABLET_DATA_DELETED (replaced by tablet 26cc2d5ccb7e49ab92c8b1da09152b27) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:25.944751 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 297945c50119432a9f9d3e356d573879 consensus state has no leader
I20251029 08:52:25.945091 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_eba74fd181d54950825081b6abd3201a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:25.945572 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eba74fd181d54950825081b6abd3201a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:25.954494 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c14c63f74f74a18902a17a4d210a90d with delete_type TABLET_DATA_DELETED (replaced by tablet 26cc2d5ccb7e49ab92c8b1da09152b27) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:25.955000 888 tablet_service.cc:1505] Processing CreateTablet for tablet 777fd02e599a43f58f0e774b7717266d (DEFAULT_TABLE table=default.Table_1b99bdda9ee74a1187811dccc1daf159 [id=0baaf2b577a842e09e8fb83db4bce406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.955454 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777fd02e599a43f58f0e774b7717266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.957660 2451 raft_consensus.cc:493] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.957790 2451 raft_consensus.cc:515] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.958328 2451 leader_election.cc:290] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:25.959048 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be4bcf9e3324bd884e946fae20648fd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:25.959213 1065 raft_consensus.cc:2468] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:25.959561 861 leader_election.cc:304] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:25.959842 861 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:25.960237 2554 raft_consensus.cc:2804] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:25.960307 2554 raft_consensus.cc:493] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:25.960371 2554 raft_consensus.cc:3060] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.961788 1183 consensus_queue.cc:260] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.962271 1183 ts_tablet_manager.cc:1434] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20251029 08:52:25.963513 1179 ts_tablet_manager.cc:1929] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.963618 1179 log.cc:1199] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/fc04616301354cfe873c825a9f5d7ab6
I20251029 08:52:25.963963 1179 ts_tablet_manager.cc:1950] T fc04616301354cfe873c825a9f5d7ab6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:25.965008 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:25.965864 1183 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:25.967939 1176 ts_tablet_manager.cc:1929] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:25.968621 1176 log.cc:1199] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/667c71a8dfd54c05ab56e845653c0c9a
I20251029 08:52:25.970412 1176 ts_tablet_manager.cc:1950] T 667c71a8dfd54c05ab56e845653c0c9a P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:25.972128 1183 tablet_bootstrap.cc:654] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:25.967265 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 924) took 3295 ms. Trace:
I20251029 08:52:25.973076 1179 rpcz_store.cc:276] 1029 08:52:22.671963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.672100 (+ 137us) service_pool.cc:225] Handling call
1029 08:52:25.927497 (+3255397us) ts_tablet_manager.cc:1151] Deleting tablet fc04616301354cfe873c825a9f5d7ab6
1029 08:52:25.927506 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.933963 (+ 6457us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.963417 (+ 29454us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.967252 (+ 3835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3255208}
I20251029 08:52:25.973789 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet fc04616301354cfe873c825a9f5d7ab6 (table default.Table_2eb98ef2bcb744d9b4560b816d57442d [id=7012a331fef74bf6b56ef43e8c847a0f]) successfully deleted
I20251029 08:52:25.974720 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 777fd02e599a43f58f0e774b7717266d (DEFAULT_TABLE table=default.Table_1b99bdda9ee74a1187811dccc1daf159 [id=0baaf2b577a842e09e8fb83db4bce406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.976171 1179 tablet_replica.cc:333] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:25.976485 890 tablet_service.cc:1505] Processing CreateTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b (DEFAULT_TABLE table=default.Table_3e8df650164e496298534433c953840e [id=4e08f07df16f4ed7922f498d536faa96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.976867 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27701fbf5b44e90ba33bed99fdc4d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.977501 1179 raft_consensus.cc:2243] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.977838 1179 raft_consensus.cc:2272] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.977237 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b (DEFAULT_TABLE table=default.Table_3e8df650164e496298534433c953840e [id=4e08f07df16f4ed7922f498d536faa96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.978885 1179 ts_tablet_manager.cc:1916] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.979197 2554 raft_consensus.cc:515] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:25.979764 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777fd02e599a43f58f0e774b7717266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.979795 2554 leader_election.cc:290] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:25.980815 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be4bcf9e3324bd884e946fae20648fd" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:25.980973 1065 raft_consensus.cc:3060] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.981195 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27701fbf5b44e90ba33bed99fdc4d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.981166 889 tablet_service.cc:1505] Processing CreateTablet for tablet ad8a88123f8542eea86d8656bfef97df (DEFAULT_TABLE table=default.Table_4b2a8ce7300d4dfda4127e313d914f22 [id=7a37f6e30a304e17a7a0ab5bd6b9564d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.981474 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad8a88123f8542eea86d8656bfef97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.983641 1065 raft_consensus.cc:2468] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:25.984097 861 leader_election.cc:304] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:25.984373 2451 raft_consensus.cc:2804] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:25.984499 2451 raft_consensus.cc:697] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:25.984721 2451 consensus_queue.cc:237] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:25.986007 861 leader_election.cc:343] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:25.987251 1974 ts_tablet_manager.cc:1929] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:25.987099 758 tablet_service.cc:1505] Processing CreateTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b (DEFAULT_TABLE table=default.Table_3e8df650164e496298534433c953840e [id=4e08f07df16f4ed7922f498d536faa96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:25.987360 1974 log.cc:1199] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/41e435e9dfb34d4f8b0c8f7b702d5a6c
I20251029 08:52:25.987571 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27701fbf5b44e90ba33bed99fdc4d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:25.987761 1974 ts_tablet_manager.cc:1950] T 41e435e9dfb34d4f8b0c8f7b702d5a6c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:25.988116 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 971) took 2995 ms. Trace:
I20251029 08:52:25.988207 1974 rpcz_store.cc:276] 1029 08:52:22.992778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.992948 (+ 170us) service_pool.cc:225] Handling call
1029 08:52:25.930128 (+2937180us) ts_tablet_manager.cc:1151] Deleting tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c
1029 08:52:25.930138 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.963881 (+ 33743us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.987175 (+ 23294us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:25.988099 (+ 924us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432,"tablet-delete.queue_time_us":2937047}
I20251029 08:52:25.988567 1974 tablet_replica.cc:333] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:25.988695 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 41e435e9dfb34d4f8b0c8f7b702d5a6c (table default.Table_d22c6f8438dc46d79b03e79db3d79cac [id=c0c3f7583c514d47b249e55073cb1300]) successfully deleted
I20251029 08:52:25.988761 1974 raft_consensus.cc:2243] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:25.989055 1974 pending_rounds.cc:70] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:25.989117 1974 pending_rounds.cc:77] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037658037243904 op_type: NO_OP noop_request { }
I20251029 08:52:25.989219 1974 raft_consensus.cc:2889] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:25.989305 1974 raft_consensus.cc:2272] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:25.989777 922 raft_consensus.cc:3060] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:25.989854 1974 ts_tablet_manager.cc:1916] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:25.990417 2636 raft_consensus.cc:493] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.990773 2636 raft_consensus.cc:515] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.991465 2603 tablet.cc:1722] T b0ff0658e0274cf49c389ad602d91b03 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:25.991555 2458 raft_consensus.cc:493] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:25.991585 2636 leader_election.cc:290] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.991696 2458 raft_consensus.cc:515] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:25.992260 2458 leader_election.cc:290] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:25.992851 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:25.993328 718 leader_election.cc:343] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.993722 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "391977048e5f4f0b8f62e4d5e3d2946b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.995461 720 leader_election.cc:343] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.995689 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "391977048e5f4f0b8f62e4d5e3d2946b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:25.995682 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:25.996124 720 leader_election.cc:343] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.996238 720 leader_election.cc:304] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:25.996346 718 leader_election.cc:343] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:25.996476 718 leader_election.cc:304] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:25.996702 2636 raft_consensus.cc:2749] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:25.996749 2458 raft_consensus.cc:2749] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.001048 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 667c71a8dfd54c05ab56e845653c0c9a (table default.Table_c9c0266a3d25476cae75216163270a5f [id=3b0c410e5d1e4d33ab2f50cff45fb590]) successfully deleted
I20251029 08:52:26.001549 753 tablet_service.cc:1505] Processing CreateTablet for tablet ad8a88123f8542eea86d8656bfef97df (DEFAULT_TABLE table=default.Table_4b2a8ce7300d4dfda4127e313d914f22 [id=7a37f6e30a304e17a7a0ab5bd6b9564d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:26.002666 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a2c13db40d854269b13340aa6acef83e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c13db40d854269b13340aa6acef83e already in progress: creating tablet
W20251029 08:52:26.004062 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b1af4bcd7af49a0b8a419298ab37ea9 already in progress: creating tablet
I20251029 08:52:26.004489 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad8a88123f8542eea86d8656bfef97df. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:26.005384 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f062556d67a148d19d242b8d99cfdff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f062556d67a148d19d242b8d99cfdff9 already in progress: creating tablet
I20251029 08:52:26.006883 1974 ts_tablet_manager.cc:1929] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 08:52:26.006944 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d0d00a6b772d4034ac7fa764884c7081 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0d00a6b772d4034ac7fa764884c7081 already in progress: creating tablet
I20251029 08:52:26.006982 1974 log.cc:1199] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/de78ea608c904aeea7f7145b9c80f45b
I20251029 08:52:26.007298 1974 ts_tablet_manager.cc:1950] T de78ea608c904aeea7f7145b9c80f45b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:26.007562 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 972) took 3014 ms. Trace:
I20251029 08:52:26.007644 1974 rpcz_store.cc:276] 1029 08:52:22.992924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.993103 (+ 179us) service_pool.cc:225] Handling call
1029 08:52:25.988510 (+2995407us) ts_tablet_manager.cc:1151] Deleting tablet de78ea608c904aeea7f7145b9c80f45b
1029 08:52:25.988523 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.003612 (+ 15089us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.006802 (+ 3190us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.007548 (+ 746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995298}
I20251029 08:52:26.007977 1974 tablet_replica.cc:333] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:26.008277 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet de78ea608c904aeea7f7145b9c80f45b (table default.Table_770bebdc34fe40eab00f06e1d53a4d06 [id=8c27488b46d84ac8903417d9c1485017]) successfully deleted
I20251029 08:52:26.009210 1035 tablet_service.cc:1505] Processing CreateTablet for tablet ad8a88123f8542eea86d8656bfef97df (DEFAULT_TABLE table=default.Table_4b2a8ce7300d4dfda4127e313d914f22 [id=7a37f6e30a304e17a7a0ab5bd6b9564d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.009577 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad8a88123f8542eea86d8656bfef97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.017086 1176 tablet_replica.cc:333] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.018455 1176 raft_consensus.cc:2243] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.020229 2366 raft_consensus.cc:493] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.020361 2366 raft_consensus.cc:515] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.020069 1176 raft_consensus.cc:2272] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.020974 2366 leader_election.cc:290] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:26.024145 998 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09968d93203140fa9201928c87702ee0
I20251029 08:52:26.026908 1176 ts_tablet_manager.cc:1916] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.033254 2636 raft_consensus.cc:493] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.033442 2636 raft_consensus.cc:515] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.034117 2636 leader_election.cc:290] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.034751 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf29e030892d44e1ad4d1ca4f5aeab3b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:26.035420 720 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.035612 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_fc24bd6f44c0439f9a83693d672608a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.036237 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc24bd6f44c0439f9a83693d672608a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.038708 2636 raft_consensus.cc:493] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.038857 2636 raft_consensus.cc:515] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.039449 2636 leader_election.cc:290] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.040091 752 tablet_service.cc:1505] Processing CreateTablet for tablet 08a617fbb0bf4d88afad59e868c609be (DEFAULT_TABLE table=default.Table_cafb2beec81f464486fab880551e8b66 [id=7d25fcc838be418bb5f271a35e753d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.040498 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08a617fbb0bf4d88afad59e868c609be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.043557 754 tablet_service.cc:1505] Processing CreateTablet for tablet f860052fec124192a67f28ffd43f9380 (DEFAULT_TABLE table=default.Table_c0ef8fb316bd46f8894cd22f35970a80 [id=ce4440afda8141709295df278003a0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.044041 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f860052fec124192a67f28ffd43f9380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.044898 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4524a248859448c5b789c7ae82435705 (DEFAULT_TABLE table=default.Table_ecb4a70de9424d629eaa74f4be4c6549 [id=272c88cc4f66469d87899f1e1062eb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.044973 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.045234 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4524a248859448c5b789c7ae82435705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.045392 889 tablet_service.cc:1505] Processing CreateTablet for tablet 4524a248859448c5b789c7ae82435705 (DEFAULT_TABLE table=default.Table_ecb4a70de9424d629eaa74f4be4c6549 [id=272c88cc4f66469d87899f1e1062eb60]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:26.045467 718 leader_election.cc:343] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.045751 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4524a248859448c5b789c7ae82435705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.047212 764 tablet_service.cc:1505] Processing CreateTablet for tablet be4e386295d64237acc5496cfa4efd75 (DEFAULT_TABLE table=default.Table_448dfa8e59de435faa660bb3e9ee4cd4 [id=06594a9864c7416aaa4d0a391946b264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.047606 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4e386295d64237acc5496cfa4efd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.048306 892 tablet_service.cc:1505] Processing CreateTablet for tablet be4e386295d64237acc5496cfa4efd75 (DEFAULT_TABLE table=default.Table_448dfa8e59de435faa660bb3e9ee4cd4 [id=06594a9864c7416aaa4d0a391946b264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.048748 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4e386295d64237acc5496cfa4efd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.048951 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.016609 890 tablet_service.cc:1505] Processing CreateTablet for tablet 08a617fbb0bf4d88afad59e868c609be (DEFAULT_TABLE table=default.Table_cafb2beec81f464486fab880551e8b66 [id=7d25fcc838be418bb5f271a35e753d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.021893 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09968d93203140fa9201928c87702ee0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.023579 1177 tablet_bootstrap.cc:492] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:26.049798 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08a617fbb0bf4d88afad59e868c609be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.049846 1177 ts_tablet_manager.cc:1403] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.135s user 0.002s sys 0.003s
I20251029 08:52:26.033464 2603 tablet.cc:1722] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:26.033650 2295 raft_consensus.cc:493] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:26.050482 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 4524a248859448c5b789c7ae82435705 (DEFAULT_TABLE table=default.Table_ecb4a70de9424d629eaa74f4be4c6549 [id=272c88cc4f66469d87899f1e1062eb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.050494 2295 raft_consensus.cc:515] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.050911 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4524a248859448c5b789c7ae82435705. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:26.052259 996 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.052446 1974 raft_consensus.cc:2243] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:26.052533 720 leader_election.cc:343] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.052620 1974 raft_consensus.cc:2272] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.052670 720 leader_election.cc:304] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.053128 1974 ts_tablet_manager.cc:1916] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.053109 996 leader_election.cc:304] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.053459 2366 raft_consensus.cc:2749] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.053800 2636 raft_consensus.cc:2749] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.053912 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 08a617fbb0bf4d88afad59e868c609be (DEFAULT_TABLE table=default.Table_cafb2beec81f464486fab880551e8b66 [id=7d25fcc838be418bb5f271a35e753d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.054260 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08a617fbb0bf4d88afad59e868c609be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.054468 1047 tablet_service.cc:1505] Processing CreateTablet for tablet f860052fec124192a67f28ffd43f9380 (DEFAULT_TABLE table=default.Table_c0ef8fb316bd46f8894cd22f35970a80 [id=ce4440afda8141709295df278003a0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.054836 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f860052fec124192a67f28ffd43f9380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.036010 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf29e030892d44e1ad4d1ca4f5aeab3b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:26.057384 718 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.057499 718 leader_election.cc:304] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.057727 2636 raft_consensus.cc:2749] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.060734 1178 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:26.060959 1178 ts_tablet_manager.cc:1403] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.198s user 0.001s sys 0.005s
I20251029 08:52:26.061813 1178 raft_consensus.cc:359] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.062076 1178 raft_consensus.cc:385] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.062191 1178 raft_consensus.cc:740] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.062494 1178 consensus_queue.cc:260] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.063107 1178 ts_tablet_manager.cc:1434] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:26.063900 1183 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.064092 1183 ts_tablet_manager.cc:1403] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.005s
I20251029 08:52:26.064266 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:26.064852 1183 raft_consensus.cc:359] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.054216 888 tablet_service.cc:1505] Processing CreateTablet for tablet f860052fec124192a67f28ffd43f9380 (DEFAULT_TABLE table=default.Table_c0ef8fb316bd46f8894cd22f35970a80 [id=ce4440afda8141709295df278003a0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.065003 1183 raft_consensus.cc:385] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.065052 1183 raft_consensus.cc:740] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.065193 1177 raft_consensus.cc:359] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.065223 1183 consensus_queue.cc:260] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.065342 1177 raft_consensus.cc:385] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.065465 1178 tablet_bootstrap.cc:492] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:26.065701 1183 ts_tablet_manager.cc:1434] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 08:52:26.065943 1183 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.067039 1178 tablet_bootstrap.cc:654] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.065389 1177 raft_consensus.cc:740] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.067564 1177 consensus_queue.cc:260] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.068106 1177 ts_tablet_manager.cc:1434] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251029 08:52:26.068364 1177 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:26.071241 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:26.075184 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 777fd02e599a43f58f0e774b7717266d consensus state has no leader
I20251029 08:52:26.075253 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f860052fec124192a67f28ffd43f9380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.075567 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3773b920e9b340ad8796f4a171e769a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:26.076934 1177 tablet_bootstrap.cc:654] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.077773 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "026eef7a3cba40d0a2c090485cea55e0" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
W20251029 08:52:26.079077 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3773b920e9b340ad8796f4a171e769a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.077294 2295 leader_election.cc:290] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.078364 861 leader_election.cc:304] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:26.082166 2295 raft_consensus.cc:2749] T 026eef7a3cba40d0a2c090485cea55e0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 08:52:26.082800 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21d409c2e24947beb99351da46a5aef9 has no consensus state
I20251029 08:52:26.083149 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5a7aba9b7fad44f899406973fdc3a841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.083658 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a7aba9b7fad44f899406973fdc3a841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.084122 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_465b5a50b6e54ff686adb35a53a61809 [id=ad2b093021204f7eaf35330fd0675654] alter complete (version 1)
W20251029 08:52:26.084683 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bda184ab18764ae592a6a3a25ef9a724 consensus state has no leader
I20251029 08:52:26.084888 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 919e5494b84343639a2b9b81b05f1c2c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:26.091315 1179 ts_tablet_manager.cc:1929] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.091441 1179 log.cc:1199] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:26.091784 1179 ts_tablet_manager.cc:1950] T bbc4345cabbc476485b6a040a076ddaf P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:26.095623 582 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6bae8ba92b214a7c99bec970f3cfb68d (table default.Table_60a71e471a9f4523a163663b4edbf15d [id=6c92cf4eb68440acb4ac55e66a7e37e4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6bae8ba92b214a7c99bec970f3cfb68d consensus state has no leader
I20251029 08:52:26.099172 590 master_service.cc:954] ReplaceTablet: received request to replace tablet eed66c3fa3f848df905a2a31fd553474 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:26.099694 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc4e52894b5141c0b0297a471a1c6eef" } modify_external_catalogs: true
I20251029 08:52:26.099843 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dc4e52894b5141c0b0297a471a1c6eef" } modify_external_catalogs: true
I20251029 08:52:26.100147 588 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_c0e6b2aa7a434d399ef4384fe8093945 [id=8b155393505643e9b1b66a0a8ac5e4f2] alter complete (version 5)
I20251029 08:52:26.101356 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_71c355ca7af84373b74ea04307f6a818" } modify_external_catalogs: true
I20251029 08:52:26.101492 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_71c355ca7af84373b74ea04307f6a818" } modify_external_catalogs: true
I20251029 08:52:26.102913 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b58caa6aed5542948934aeca3e67d32d" } modify_external_catalogs: true
I20251029 08:52:26.103049 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b58caa6aed5542948934aeca3e67d32d" } modify_external_catalogs: true
I20251029 08:52:26.103068 1176 ts_tablet_manager.cc:1929] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:26.103269 1176 log.cc:1199] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:26.103950 1176 ts_tablet_manager.cc:1950] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:26.104445 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c57b7890d1f4c8e8e9801e845e4df11" } modify_external_catalogs: true
I20251029 08:52:26.104563 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7c57b7890d1f4c8e8e9801e845e4df11" } modify_external_catalogs: true
I20251029 08:52:26.105544 1176 tablet_replica.cc:333] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.105829 1176 raft_consensus.cc:2243] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.106069 1176 raft_consensus.cc:2272] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.106936 1176 ts_tablet_manager.cc:1916] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.108556 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2c072545dc7446eabcdbe80f548c5df7" } modify_external_catalogs: true
I20251029 08:52:26.108680 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2c072545dc7446eabcdbe80f548c5df7" } modify_external_catalogs: true
I20251029 08:52:26.111261 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 63e705a0b5264eb7a6988129edbf5970 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:26.114886 2366 raft_consensus.cc:493] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.117198 1176 ts_tablet_manager.cc:1929] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:26.117499 1176 log.cc:1199] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2fb380018f46465e82ac6223ecf8dd0d
I20251029 08:52:26.118249 1176 ts_tablet_manager.cc:1950] T 2fb380018f46465e82ac6223ecf8dd0d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:26.119837 1176 tablet_replica.cc:333] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.120278 1176 raft_consensus.cc:2243] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.120620 1176 raft_consensus.cc:2272] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.120656 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 39fc9612689b46d498d6092bd552da4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.120936 902 tablet_service.cc:1505] Processing CreateTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.121424 895 tablet_service.cc:1505] Processing CreateTablet for tablet 91622c8b93ac40679e1fea797838ea99 (DEFAULT_TABLE table=default.Table_16e768ada8754f51a74b4d88207a0ddf [id=2f13880e2e4f4dedb0bddea643ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.121507 886 tablet_service.cc:1505] Processing CreateTablet for tablet 908d97d9d82d49bbaaf6af372877c920 (DEFAULT_TABLE table=default.Table_875a944d09b24fb98857e00795b8e64e [id=8b39c6ace15e4381af7eaf34c288ad3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.121732 1176 ts_tablet_manager.cc:1916] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.121824 894 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52a754aca0467d9733107685a3bca2 (DEFAULT_TABLE table=default.Table_5fd1636a22104e95a64e29baa9584e3c [id=eb770139ec0f4ce4be7c04e04cbf6b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.121850 887 tablet_service.cc:1552] Processing DeleteTablet for tablet e8c4d793f10147fba5e4f44474575bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a806cc04e641d9a1c4af97a24b13de) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.122120 891 tablet_service.cc:1505] Processing CreateTablet for tablet f6b4ac5e43734692b775fe50f760a832 (DEFAULT_TABLE table=default.Table_1a0289f2e27f44a3a7089b80c844d2d2 [id=dfd4e167d48f4168ad6c9f651251a85a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.122157 897 tablet_service.cc:1505] Processing CreateTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 (DEFAULT_TABLE table=default.Table_588b8eaf2cad4b739475473bec43d9cf [id=be7a87faae7a4ee5afef9c181e296229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.121869 899 tablet_service.cc:1505] Processing CreateTablet for tablet c7af99e95e334279ac160ca7e0b85e77 (DEFAULT_TABLE table=default.Table_5d3db711683243eabfce1d1543d63f9f [id=f54d5db45b5844a9b52c55ce3b9236ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.122619 901 tablet_service.cc:1505] Processing CreateTablet for tablet f284bc30fd1144d69e0e11ea648c5ed7 (DEFAULT_TABLE table=default.Table_47b07a98fcc34ec2a2784a43b313f904 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.120936 896 tablet_service.cc:1505] Processing CreateTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.123085 893 tablet_service.cc:1505] Processing CreateTablet for tablet 06b04fdc8080474fbc92fc5606a41c00 (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.123567 904 tablet_service.cc:1505] Processing CreateTablet for tablet adba5919a81348bb94569ca52fa4bded (DEFAULT_TABLE table=default.Table_5ec3d23d22464070b4557777a2306288 [id=20bf94accf0e499c9868ba2fb08c9690]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.123992 898 tablet_service.cc:1505] Processing CreateTablet for tablet f004f284a60346dba6252ecd3daf5698 (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.130420 753 tablet_service.cc:1505] Processing CreateTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.130810 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b611ae08a247d3ba02eb177c6b1b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.131347 749 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.131508 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]) successfully deleted
I20251029 08:52:26.131616 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.131701 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2fb380018f46465e82ac6223ecf8dd0d (table default.Table_1d27485babbd4ce5afd764a5d187d095 [id=ccc56f591675423da2206c023b1f8853]) successfully deleted
I20251029 08:52:26.131870 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a73a891fb6034f989d29a3f837303d84 with delete_type TABLET_DATA_DELETED (replaced by tablet a711e95fbef84b12997d01c1883057d9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.132129 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.132416 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.132804 749 tablet_service.cc:1505] Processing CreateTablet for tablet f284bc30fd1144d69e0e11ea648c5ed7 (DEFAULT_TABLE table=default.Table_47b07a98fcc34ec2a2784a43b313f904 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.133010 754 tablet_service.cc:1505] Processing CreateTablet for tablet 06b04fdc8080474fbc92fc5606a41c00 (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.133143 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f284bc30fd1144d69e0e11ea648c5ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.133275 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b04fdc8080474fbc92fc5606a41c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.133764 892 tablet_service.cc:1552] Processing DeleteTablet for tablet bbc4345cabbc476485b6a040a076ddaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.134491 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.135571 752 tablet_service.cc:1505] Processing CreateTablet for tablet 91622c8b93ac40679e1fea797838ea99 (DEFAULT_TABLE table=default.Table_16e768ada8754f51a74b4d88207a0ddf [id=2f13880e2e4f4dedb0bddea643ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.135921 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91622c8b93ac40679e1fea797838ea99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.136615 745 tablet_service.cc:1505] Processing CreateTablet for tablet c7af99e95e334279ac160ca7e0b85e77 (DEFAULT_TABLE table=default.Table_5d3db711683243eabfce1d1543d63f9f [id=f54d5db45b5844a9b52c55ce3b9236ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.136929 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7af99e95e334279ac160ca7e0b85e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.137275 757 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52a754aca0467d9733107685a3bca2 (DEFAULT_TABLE table=default.Table_5fd1636a22104e95a64e29baa9584e3c [id=eb770139ec0f4ce4be7c04e04cbf6b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.137594 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52a754aca0467d9733107685a3bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.140244 746 tablet_service.cc:1505] Processing CreateTablet for tablet adba5919a81348bb94569ca52fa4bded (DEFAULT_TABLE table=default.Table_5ec3d23d22464070b4557777a2306288 [id=20bf94accf0e499c9868ba2fb08c9690]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.140606 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adba5919a81348bb94569ca52fa4bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.140767 1035 tablet_service.cc:1505] Processing CreateTablet for tablet be4e386295d64237acc5496cfa4efd75 (DEFAULT_TABLE table=default.Table_448dfa8e59de435faa660bb3e9ee4cd4 [id=06594a9864c7416aaa4d0a391946b264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.141113 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4e386295d64237acc5496cfa4efd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.143770 750 tablet_service.cc:1505] Processing CreateTablet for tablet f004f284a60346dba6252ecd3daf5698 (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.140251 747 tablet_service.cc:1505] Processing CreateTablet for tablet 908d97d9d82d49bbaaf6af372877c920 (DEFAULT_TABLE table=default.Table_875a944d09b24fb98857e00795b8e64e [id=8b39c6ace15e4381af7eaf34c288ad3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.144490 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908d97d9d82d49bbaaf6af372877c920. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:26.145285 998 consensus_peers.cc:597] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8c4d57a59684723bfd27c7dda302ef3. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.145517 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f004f284a60346dba6252ecd3daf5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.157954 764 tablet_service.cc:1505] Processing CreateTablet for tablet f6b4ac5e43734692b775fe50f760a832 (DEFAULT_TABLE table=default.Table_1a0289f2e27f44a3a7089b80c844d2d2 [id=dfd4e167d48f4168ad6c9f651251a85a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:26.161898 998 consensus_peers.cc:597] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:26.163341 861 consensus_peers.cc:597] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39fc9612689b46d498d6092bd552da4e. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.164799 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b4ac5e43734692b775fe50f760a832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.168900 1045 tablet_service.cc:1505] Processing CreateTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.171522 1041 tablet_service.cc:1505] Processing CreateTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.169198 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.176630 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 91622c8b93ac40679e1fea797838ea99 (DEFAULT_TABLE table=default.Table_16e768ada8754f51a74b4d88207a0ddf [id=2f13880e2e4f4dedb0bddea643ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.177029 1049 tablet_service.cc:1505] Processing CreateTablet for tablet f6b4ac5e43734692b775fe50f760a832 (DEFAULT_TABLE table=default.Table_1a0289f2e27f44a3a7089b80c844d2d2 [id=dfd4e167d48f4168ad6c9f651251a85a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.177078 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 925) took 3497 ms. Trace:
I20251029 08:52:26.177377 1179 rpcz_store.cc:276] 1029 08:52:22.679464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.679516 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:25.973466 (+3293950us) ts_tablet_manager.cc:1151] Deleting tablet bbc4345cabbc476485b6a040a076ddaf
1029 08:52:25.976121 (+ 2655us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:25.983389 (+ 7268us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.091216 (+107827us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.177005 (+ 85789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3293774}
I20251029 08:52:26.115032 2366 raft_consensus.cc:515] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.180625 2366 leader_election.cc:290] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.181367 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.182960 747 tablet_service.cc:1505] Processing CreateTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 (DEFAULT_TABLE table=default.Table_588b8eaf2cad4b739475473bec43d9cf [id=be7a87faae7a4ee5afef9c181e296229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.184170 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 919e5494b84343639a2b9b81b05f1c2c
I20251029 08:52:26.184646 591 catalog_manager.cc:6719] ReplaceTablet: tablet 919e5494b84343639a2b9b81b05f1c2c deleted and replaced by tablet adbff30ec60c4e559f95ddb21b60e232
I20251029 08:52:26.184942 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d2bfdd8e87004d21a2742e8b4780a151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.185472 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2bfdd8e87004d21a2742e8b4780a151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.188457 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 919e5494b84343639a2b9b81b05f1c2c with delete_type TABLET_DATA_DELETED (replaced by tablet adbff30ec60c4e559f95ddb21b60e232) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.189213 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 919e5494b84343639a2b9b81b05f1c2c with delete_type TABLET_DATA_DELETED (replaced by tablet adbff30ec60c4e559f95ddb21b60e232) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:26.189617 998 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24834985a71349b9865840e9b5537d72
I20251029 08:52:26.190253 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ebb42f8e232e40eca333dd1e8b60cc62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.190791 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebb42f8e232e40eca333dd1e8b60cc62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.191932 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b611ae08a247d3ba02eb177c6b1b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.193346 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6472d3c59db1490eb91b81a60372f0d4" } new_table_name: "default.Table_9c229240a45f4c07878177a7c961d5f0" modify_external_catalogs: true
I20251029 08:52:26.194887 2366 raft_consensus.cc:493] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.195020 2366 raft_consensus.cc:515] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.195587 2366 leader_election.cc:290] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.200594 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3c9cdcec8e494050a475defd0cfd119f
I20251029 08:52:26.200524 750 tablet_service.cc:1505] Processing CreateTablet for tablet 0f9b0252b432464e8c07204a189bd619 (DEFAULT_TABLE table=default.Table_c385a4409927408eb2a8e027b4eeba95 [id=d2542c8264c64af2889a5290bc6bd9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.200592 1047 tablet_service.cc:1505] Processing CreateTablet for tablet f284bc30fd1144d69e0e11ea648c5ed7 (DEFAULT_TABLE table=default.Table_47b07a98fcc34ec2a2784a43b313f904 [id=0265c5ac448a46a49a1bfcca4a305645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.201553 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9cdcec8e494050a475defd0cfd119f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:26.201683 998 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:26.202013 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9cdcec8e494050a475defd0cfd119f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.203366 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 06b04fdc8080474fbc92fc5606a41c00 (DEFAULT_TABLE table=default.Table_b055a666e8cc467c861649776a8021e0 [id=995a1634c65c42b8867fa7922c7cee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.204341 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 908d97d9d82d49bbaaf6af372877c920 (DEFAULT_TABLE table=default.Table_875a944d09b24fb98857e00795b8e64e [id=8b39c6ace15e4381af7eaf34c288ad3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.204305 892 tablet_service.cc:1505] Processing CreateTablet for tablet 0f9b0252b432464e8c07204a189bd619 (DEFAULT_TABLE table=default.Table_c385a4409927408eb2a8e027b4eeba95 [id=d2542c8264c64af2889a5290bc6bd9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.205619 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 7a52a754aca0467d9733107685a3bca2 (DEFAULT_TABLE table=default.Table_5fd1636a22104e95a64e29baa9584e3c [id=eb770139ec0f4ce4be7c04e04cbf6b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.205844 1050 tablet_service.cc:1505] Processing CreateTablet for tablet f004f284a60346dba6252ecd3daf5698 (DEFAULT_TABLE table=default.Table_655b57319ec4484288b2c34c169145ca [id=a1e46bdbb3fd43608ff11aa32f3bb5af]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:26.205938 718 consensus_peers.cc:597] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.207118 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3e8df650164e496298534433c953840e" } new_table_name: "default.Table_d868be96231b420db67789c6ba1d5d2a" modify_external_catalogs: true
I20251029 08:52:26.209415 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c7af99e95e334279ac160ca7e0b85e77 (DEFAULT_TABLE table=default.Table_5d3db711683243eabfce1d1543d63f9f [id=f54d5db45b5844a9b52c55ce3b9236ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.210130 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 919e5494b84343639a2b9b81b05f1c2c with delete_type TABLET_DATA_DELETED (replaced by tablet adbff30ec60c4e559f95ddb21b60e232) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.210238 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9b0252b432464e8c07204a189bd619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.210768 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c9cdcec8e494050a475defd0cfd119f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.210870 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960f17ab6bed4ef5ad382421d84bf8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.210855 1039 tablet_service.cc:1505] Processing CreateTablet for tablet adba5919a81348bb94569ca52fa4bded (DEFAULT_TABLE table=default.Table_5ec3d23d22464070b4557777a2306288 [id=20bf94accf0e499c9868ba2fb08c9690]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.211140 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 (DEFAULT_TABLE table=default.Table_588b8eaf2cad4b739475473bec43d9cf [id=be7a87faae7a4ee5afef9c181e296229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.211728 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 0f9b0252b432464e8c07204a189bd619 (DEFAULT_TABLE table=default.Table_c385a4409927408eb2a8e027b4eeba95 [id=d2542c8264c64af2889a5290bc6bd9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.212553 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bbc4345cabbc476485b6a040a076ddaf (table default.Table_64f9de5287d54b348a406a42a7c379ab [id=0e01efd35bcd43dc88e2d92f0734b958]) successfully deleted
I20251029 08:52:26.213819 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24834985a71349b9865840e9b5537d72" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.214756 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fdc381a0d949b0b93cef399daa3021" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.214397 748 tablet_service.cc:1505] Processing CreateTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.215843 890 tablet_service.cc:1505] Processing CreateTablet for tablet f195194e3efa48d6937497b842e7f528 (DEFAULT_TABLE table=default.Table_0336e0b703a5476bbae10859daeee070 [id=e4bccd14de8c4588a5888e916d8cca8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.217406 2475 consensus_queue.cc:1048] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.218220 2359 consensus_queue.cc:1048] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251029 08:52:26.218004 996 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.220589 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac
I20251029 08:52:26.222081 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.223892 2295 consensus_queue.cc:1048] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.224573 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4c77960427da4195879204c4bd7833a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.225102 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c77960427da4195879204c4bd7833a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.225678 745 tablet_service.cc:1505] Processing CreateTablet for tablet f195194e3efa48d6937497b842e7f528 (DEFAULT_TABLE table=default.Table_0336e0b703a5476bbae10859daeee070 [id=e4bccd14de8c4588a5888e916d8cca8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.228435 2611 tablet.cc:1722] T 713e102d9e1d40148636443a2260ccbc P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:26.230146 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9adbcf3f12104fb5bfd1d9dd5c946c54
I20251029 08:52:26.232079 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.234083 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 59ad04f57c29407b9d2420520c095a28
I20251029 08:52:26.236698 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 59ad04f57c29407b9d2420520c095a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.237372 887 tablet_service.cc:1505] Processing CreateTablet for tablet adcb69c6f0ea4052895aa9270684d7ae (DEFAULT_TABLE table=default.Table_bfb8a865be784f65ba33fe86cd9ee27e [id=42e7d69322bb444e8827f21be7c34f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.241770 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.241977 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 59ad04f57c29407b9d2420520c095a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.243290 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5c1854f1ed1c420bb4402bc4e8d53b42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.243875 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c1854f1ed1c420bb4402bc4e8d53b42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.244221 2545 tablet.cc:1722] T 713e102d9e1d40148636443a2260ccbc P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:26.246641 752 tablet_service.cc:1505] Processing CreateTablet for tablet adcb69c6f0ea4052895aa9270684d7ae (DEFAULT_TABLE table=default.Table_bfb8a865be784f65ba33fe86cd9ee27e [id=42e7d69322bb444e8827f21be7c34f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.254191 595 catalog_manager.cc:5649] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "927689e0dc5e4bc79d4a9d79a873f088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:26.254647 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:26.255087 996 leader_election.cc:304] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.255313 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c9ed0a270b794e469d2fdbd0ea1d96f8
I20251029 08:52:26.256793 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.256953 905 tablet_service.cc:1552] Processing DeleteTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:26.259261 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:26.259662 2366 raft_consensus.cc:2749] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:26.260183 996 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.260375 905 tablet_service.cc:1505] Processing CreateTablet for tablet 6bae8ba92b214a7c99bec970f3cfb68d (DEFAULT_TABLE table=default.Table_60a71e471a9f4523a163663b4edbf15d [id=6c92cf4eb68440acb4ac55e66a7e37e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.260893 996 leader_election.cc:304] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.261399 2366 raft_consensus.cc:2749] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.269655 757 tablet_service.cc:1505] Processing CreateTablet for tablet 6bae8ba92b214a7c99bec970f3cfb68d (DEFAULT_TABLE table=default.Table_60a71e471a9f4523a163663b4edbf15d [id=6c92cf4eb68440acb4ac55e66a7e37e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.272817 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c18e4c41714ba38bdd2e208a5e761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.273028 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b4ac5e43734692b775fe50f760a832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.273288 2636 raft_consensus.cc:493] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.273459 2636 raft_consensus.cc:515] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.274070 2636 leader_election.cc:290] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.274502 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657cdaad97e24611a997e26a6293ea5e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.276196 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91622c8b93ac40679e1fea797838ea99. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:26.276783 718 leader_election.cc:343] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.278278 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_af4260159d7346c49aabb31cfa9f224f" } new_table_name: "default.Table_f9cf539740484536bda2e00dc00f7a76" modify_external_catalogs: true
I20251029 08:52:26.282140 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 926) took 3524 ms. Trace:
I20251029 08:52:26.282269 1179 rpcz_store.cc:276] 1029 08:52:22.758136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.758192 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:26.178399 (+3420207us) ts_tablet_manager.cc:1151] Deleting tablet 5c93076ca44542a5ae19e2c73dba95cb
1029 08:52:26.281993 (+103594us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.282122 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3348436}
I20251029 08:52:26.282634 1179 tablet_replica.cc:333] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.282872 1179 raft_consensus.cc:2243] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:26.283274 1179 raft_consensus.cc:2272] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.283875 1179 ts_tablet_manager.cc:1916] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.284328 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b04fdc8080474fbc92fc5606a41c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.284509 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f284bc30fd1144d69e0e11ea648c5ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.284782 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908d97d9d82d49bbaaf6af372877c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.285126 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52a754aca0467d9733107685a3bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.288242 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f004f284a60346dba6252ecd3daf5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.288782 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d9b9417a07e84ab38c3498a14cbbb03d" } new_table_name: "default.Table_e2619014a196457894632245cfd2d8bc" modify_external_catalogs: true
I20251029 08:52:26.294741 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7af99e95e334279ac160ca7e0b85e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.295678 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adba5919a81348bb94569ca52fa4bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.300128 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960f17ab6bed4ef5ad382421d84bf8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.301455 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9b0252b432464e8c07204a189bd619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.301769 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb69c6f0ea4052895aa9270684d7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.302062 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f195194e3efa48d6937497b842e7f528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.303448 1177 tablet_bootstrap.cc:492] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:26.303592 1177 ts_tablet_manager.cc:1403] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.235s user 0.005s sys 0.001s
I20251029 08:52:26.304152 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c18e4c41714ba38bdd2e208a5e761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.304378 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae8ba92b214a7c99bec970f3cfb68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.306262 1183 tablet_bootstrap.cc:654] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.326103 1176 ts_tablet_manager.cc:1929] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:26.326710 1176 log.cc:1199] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/6689cf0804f946c18ebc555a191549d4
I20251029 08:52:26.332396 1176 ts_tablet_manager.cc:1950] T 6689cf0804f946c18ebc555a191549d4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:26.326488 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.324793 1177 raft_consensus.cc:359] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.336767 1177 raft_consensus.cc:385] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.336947 1177 raft_consensus.cc:740] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.337357 1177 consensus_queue.cc:260] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.338392 1177 ts_tablet_manager.cc:1434] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20251029 08:52:26.342748 1044 tablet_service.cc:1505] Processing CreateTablet for tablet adcb69c6f0ea4052895aa9270684d7ae (DEFAULT_TABLE table=default.Table_bfb8a865be784f65ba33fe86cd9ee27e [id=42e7d69322bb444e8827f21be7c34f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.343168 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb69c6f0ea4052895aa9270684d7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.343539 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 6bae8ba92b214a7c99bec970f3cfb68d (DEFAULT_TABLE table=default.Table_60a71e471a9f4523a163663b4edbf15d [id=6c92cf4eb68440acb4ac55e66a7e37e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:26.343737 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5c93076ca44542a5ae19e2c73dba95cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c93076ca44542a5ae19e2c73dba95cb already in progress: creating tablet
I20251029 08:52:26.343863 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae8ba92b214a7c99bec970f3cfb68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.348476 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c9ed0a270b794e469d2fdbd0ea1d96f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.357249 748 tablet_service.cc:1505] Processing CreateTablet for tablet f9ea32f5eecf41a39925cdd5c07f9a88 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.357725 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ea32f5eecf41a39925cdd5c07f9a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.363981 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 6689cf0804f946c18ebc555a191549d4 (table default.Table_a565b344ef8e4b60b07cb259f877df2a [id=bef1fc3338f941f18e6ce2019247b13e]) successfully deleted
I20251029 08:52:26.366184 1177 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:26.367573 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0c2072c16fce43278f1e32b9672187e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c2072c16fce43278f1e32b9672187e7 already in progress: creating tablet
W20251029 08:52:26.371287 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 41622daeb1e0419fa88cc26c0405c2db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41622daeb1e0419fa88cc26c0405c2db already in progress: creating tablet
W20251029 08:52:26.374281 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5498826a9afd4fd5a6b88b0a465fff40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5498826a9afd4fd5a6b88b0a465fff40 already in progress: creating tablet
W20251029 08:52:26.377774 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb9ab76dcfe486e89bbe9391cab56f9 already in progress: creating tablet
W20251029 08:52:26.385363 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 52b697b962f849eb88741c2e2ba4643d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52b697b962f849eb88741c2e2ba4643d already in progress: creating tablet
I20251029 08:52:26.386123 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6b119e9386dc434daa592bd88ede248b (DEFAULT_TABLE table=default.Table_1ca0aca0e954427d84908b738587abdf [id=fd3016f76c1c49588369fb975dd2bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.387382 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae8ba92b214a7c99bec970f3cfb68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.387559 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb69c6f0ea4052895aa9270684d7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.387786 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9b0252b432464e8c07204a189bd619. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.388074 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b119e9386dc434daa592bd88ede248b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.388221 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b4ac5e43734692b775fe50f760a832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.392113 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f284bc30fd1144d69e0e11ea648c5ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.392266 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f195194e3efa48d6937497b842e7f528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.392428 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b04fdc8080474fbc92fc5606a41c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.398367 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adba5919a81348bb94569ca52fa4bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.399041 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908d97d9d82d49bbaaf6af372877c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.399883 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960f17ab6bed4ef5ad382421d84bf8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.401541 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a52a754aca0467d9733107685a3bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.406445 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c18e4c41714ba38bdd2e208a5e761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.408124 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f004f284a60346dba6252ecd3daf5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.384891 889 tablet_service.cc:1505] Processing CreateTablet for tablet f9ea32f5eecf41a39925cdd5c07f9a88 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.412396 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ea32f5eecf41a39925cdd5c07f9a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.420457 1177 tablet_bootstrap.cc:654] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.420419 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657cdaad97e24611a997e26a6293ea5e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.420651 1067 raft_consensus.cc:2468] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
W20251029 08:52:26.421900 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 58b02839973e4f018a1a6dfccf009b25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58b02839973e4f018a1a6dfccf009b25 already in progress: creating tablet
I20251029 08:52:26.422456 1178 tablet_bootstrap.cc:492] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:26.422658 1178 ts_tablet_manager.cc:1403] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.357s user 0.004s sys 0.016s
I20251029 08:52:26.423445 720 leader_election.cc:304] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.423861 2636 raft_consensus.cc:2804] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.423939 2636 raft_consensus.cc:493] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.424000 2636 raft_consensus.cc:3060] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.424635 1042 tablet_service.cc:1505] Processing CreateTablet for tablet f9ea32f5eecf41a39925cdd5c07f9a88 (DEFAULT_TABLE table=default.Table_e37db8e2538e4047b9bb40e72d99c616 [id=5aa44b809f2e4f40b2f42326c478a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.425130 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ea32f5eecf41a39925cdd5c07f9a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.426088 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:26.427762 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0c14c63f74f74a18902a17a4d210a90d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c14c63f74f74a18902a17a4d210a90d already in progress: creating tablet
I20251029 08:52:26.427439 1046 tablet_service.cc:1505] Processing CreateTablet for tablet f195194e3efa48d6937497b842e7f528 (DEFAULT_TABLE table=default.Table_0336e0b703a5476bbae10859daeee070 [id=e4bccd14de8c4588a5888e916d8cca8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.428920 2451 consensus_queue.cc:1048] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.429103 1176 tablet_replica.cc:333] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.429751 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f195194e3efa48d6937497b842e7f528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.430538 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b611ae08a247d3ba02eb177c6b1b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.423440 1178 raft_consensus.cc:359] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.434497 1178 raft_consensus.cc:385] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.434617 1178 raft_consensus.cc:740] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.434923 1178 consensus_queue.cc:260] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.436129 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91622c8b93ac40679e1fea797838ea99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.438045 1178 ts_tablet_manager.cc:1434] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251029 08:52:26.438483 1178 tablet_bootstrap.cc:492] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:26.440192 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7af99e95e334279ac160ca7e0b85e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.441944 2458 raft_consensus.cc:493] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.442063 2458 raft_consensus.cc:515] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.442688 2458 leader_election.cc:290] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.444684 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c24c0d32f74435bd9d28f2415e2c47" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:26.445259 718 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.445994 1183 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.446125 1183 ts_tablet_manager.cc:1403] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.380s user 0.001s sys 0.003s
I20251029 08:52:26.446800 1183 raft_consensus.cc:359] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.446959 1183 raft_consensus.cc:385] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.447016 1183 raft_consensus.cc:740] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.447577 1183 consensus_queue.cc:260] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.448137 1176 raft_consensus.cc:2243] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.448307 1176 raft_consensus.cc:2272] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.448844 1176 ts_tablet_manager.cc:1916] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:26.450327 861 consensus_peers.cc:597] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10baa0547180493f872a23614cf4c608. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.453866 750 tablet_service.cc:1505] Processing CreateTablet for tablet 6b119e9386dc434daa592bd88ede248b (DEFAULT_TABLE table=default.Table_1ca0aca0e954427d84908b738587abdf [id=fd3016f76c1c49588369fb975dd2bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.454238 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b119e9386dc434daa592bd88ede248b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.458801 893 tablet_service.cc:1505] Processing CreateTablet for tablet c073c4affc834fe99b85b14dc08ff531 (DEFAULT_TABLE table=default.Table_96c75b14349a481184cd42dd8b8ede6b [id=2ba1dfb4aed847269df841a549fd997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.459381 2458 raft_consensus.cc:493] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.459499 2458 raft_consensus.cc:515] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.460112 2458 leader_election.cc:290] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.460309 752 tablet_service.cc:1505] Processing CreateTablet for tablet c073c4affc834fe99b85b14dc08ff531 (DEFAULT_TABLE table=default.Table_96c75b14349a481184cd42dd8b8ede6b [id=2ba1dfb4aed847269df841a549fd997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.460548 747 tablet_service.cc:1505] Processing CreateTablet for tablet 8a8accf8f9eb4e15914287e49cc8180f (DEFAULT_TABLE table=default.Table_70588860077547cabad70977368b56fc [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.460651 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c073c4affc834fe99b85b14dc08ff531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.460640 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.460819 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8accf8f9eb4e15914287e49cc8180f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.461284 894 tablet_service.cc:1505] Processing CreateTablet for tablet 47cba03043ad44bbbed0f90013722bb4 (DEFAULT_TABLE table=default.Table_9ebc2ba31e644183b47973a987d1d9b5 [id=3031b3cae39d4fa183a7d44099ae7967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.463665 1179 ts_tablet_manager.cc:1929] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.463778 1179 log.cc:1199] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aafa846cb2144cec9ce4b28e61d91961
W20251029 08:52:26.464318 718 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.464891 898 tablet_service.cc:1505] Processing CreateTablet for tablet 8a8accf8f9eb4e15914287e49cc8180f (DEFAULT_TABLE table=default.Table_70588860077547cabad70977368b56fc [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.465399 2451 consensus_queue.cc:1048] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.467384 2636 raft_consensus.cc:515] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.467959 2636 leader_election.cc:290] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.468736 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657cdaad97e24611a997e26a6293ea5e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:26.469612 718 leader_election.cc:343] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:26.470185 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 45701.7us
I20251029 08:52:26.470520 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 59ad04f57c29407b9d2420520c095a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:26.473007 1178 tablet_bootstrap.cc:654] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:26.476864 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e2554e46fb44660893e95a0654306e9 consensus state has no leader
I20251029 08:52:26.482708 749 tablet_service.cc:1505] Processing CreateTablet for tablet 47cba03043ad44bbbed0f90013722bb4 (DEFAULT_TABLE table=default.Table_9ebc2ba31e644183b47973a987d1d9b5 [id=3031b3cae39d4fa183a7d44099ae7967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.484107 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba03043ad44bbbed0f90013722bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.488947 2636 raft_consensus.cc:493] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.489075 2636 raft_consensus.cc:515] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.489779 2636 leader_election.cc:290] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:26.490413 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cd2e9a95b6c40be9e17397a313d249b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:26.490926 718 leader_election.cc:343] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.492101 1183 ts_tablet_manager.cc:1434] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.046s user 0.001s sys 0.002s
I20251029 08:52:26.492409 1183 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.493314 750 tablet_service.cc:1505] Processing CreateTablet for tablet 74d9d97aaf2249d99ca32df777178459 (DEFAULT_TABLE table=default.Table_4a2a813a43cf444ab957477fa46cb928 [id=2e714cc376ed409aa252a15ec136ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.493867 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d9d97aaf2249d99ca32df777178459. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.493877 1183 tablet_bootstrap.cc:654] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.493849 896 tablet_service.cc:1505] Processing CreateTablet for tablet 74d9d97aaf2249d99ca32df777178459 (DEFAULT_TABLE table=default.Table_4a2a813a43cf444ab957477fa46cb928 [id=2e714cc376ed409aa252a15ec136ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.498875 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c24c0d32f74435bd9d28f2415e2c47" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.499368 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.499912 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657cdaad97e24611a997e26a6293ea5e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:26.500063 1068 raft_consensus.cc:3060] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.500882 897 tablet_service.cc:1505] Processing CreateTablet for tablet 3f25fe685db94f3ba2003bc7e417277c (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.502633 1177 tablet_bootstrap.cc:492] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:26.502833 1177 ts_tablet_manager.cc:1403] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.137s user 0.006s sys 0.000s
I20251029 08:52:26.502993 1068 raft_consensus.cc:2468] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:26.504212 758 tablet_service.cc:1505] Processing CreateTablet for tablet 3f25fe685db94f3ba2003bc7e417277c (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.504876 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f25fe685db94f3ba2003bc7e417277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.510617 747 tablet_service.cc:1505] Processing CreateTablet for tablet 04e64e5cf11547c1bfd6ac19b122efe4 (DEFAULT_TABLE table=default.Table_c204b45d751646c188724bd56add16c5 [id=06aad6e8b0ed4763857b806ff3aa4fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.511018 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e64e5cf11547c1bfd6ac19b122efe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.513568 901 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb9c8c7286149128c2741635c242341 (DEFAULT_TABLE table=default.Table_096667619d1847a998aa237bb19f9024 [id=3bea892bcc4b444d868702d8156aebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.484541 1178 tablet_bootstrap.cc:492] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:26.520895 905 tablet_service.cc:1505] Processing CreateTablet for tablet 04e64e5cf11547c1bfd6ac19b122efe4 (DEFAULT_TABLE table=default.Table_c204b45d751646c188724bd56add16c5 [id=06aad6e8b0ed4763857b806ff3aa4fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.521158 2554 consensus_queue.cc:1048] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.521871 745 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb9c8c7286149128c2741635c242341 (DEFAULT_TABLE table=default.Table_096667619d1847a998aa237bb19f9024 [id=3bea892bcc4b444d868702d8156aebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.522277 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb9c8c7286149128c2741635c242341. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.524863 1177 raft_consensus.cc:359] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.525101 1177 raft_consensus.cc:385] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.525199 1177 raft_consensus.cc:740] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.525635 1177 consensus_queue.cc:260] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.526296 1177 ts_tablet_manager.cc:1434] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
W20251029 08:52:26.532090 861 consensus_peers.cc:597] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627bc8ff99394fa2abcca15d62e5ea10. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:26.532902 998 consensus_peers.cc:597] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.534354 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cd2e9a95b6c40be9e17397a313d249b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.535764 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 8a8accf8f9eb4e15914287e49cc8180f (DEFAULT_TABLE table=default.Table_70588860077547cabad70977368b56fc [id=b07ebc517dd045a48caa07814f5c0580]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.535799 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 74d9d97aaf2249d99ca32df777178459 (DEFAULT_TABLE table=default.Table_4a2a813a43cf444ab957477fa46cb928 [id=2e714cc376ed409aa252a15ec136ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.535799 1047 tablet_service.cc:1505] Processing CreateTablet for tablet c073c4affc834fe99b85b14dc08ff531 (DEFAULT_TABLE table=default.Table_96c75b14349a481184cd42dd8b8ede6b [id=2ba1dfb4aed847269df841a549fd997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.536139 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8accf8f9eb4e15914287e49cc8180f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.536319 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d9d97aaf2249d99ca32df777178459. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.536579 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 47cba03043ad44bbbed0f90013722bb4 (DEFAULT_TABLE table=default.Table_9ebc2ba31e644183b47973a987d1d9b5 [id=3031b3cae39d4fa183a7d44099ae7967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.536918 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba03043ad44bbbed0f90013722bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.539005 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c073c4affc834fe99b85b14dc08ff531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.539960 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 6b119e9386dc434daa592bd88ede248b (DEFAULT_TABLE table=default.Table_1ca0aca0e954427d84908b738587abdf [id=fd3016f76c1c49588369fb975dd2bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.540334 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b119e9386dc434daa592bd88ede248b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.543959 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 3f25fe685db94f3ba2003bc7e417277c (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.544324 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f25fe685db94f3ba2003bc7e417277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.546804 1178 ts_tablet_manager.cc:1403] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.000s
I20251029 08:52:26.547669 1178 raft_consensus.cc:359] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.548094 1178 raft_consensus.cc:385] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.556828 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c073c4affc834fe99b85b14dc08ff531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.557096 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cba03043ad44bbbed0f90013722bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.560271 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8accf8f9eb4e15914287e49cc8180f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.560463 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f25fe685db94f3ba2003bc7e417277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.560789 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb9c8c7286149128c2741635c242341. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.571053 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e64e5cf11547c1bfd6ac19b122efe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.571396 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d9d97aaf2249d99ca32df777178459. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.576705 1177 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:26.576836 1179 ts_tablet_manager.cc:1950] T aafa846cb2144cec9ce4b28e61d91961 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:26.577155 2476 raft_consensus.cc:493] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.577303 2476 raft_consensus.cc:515] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.577929 2476 leader_election.cc:290] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.577157 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 932) took 3702 ms. Trace:
W20251029 08:52:26.578737 998 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:26.578720 1179 rpcz_store.cc:276] 1029 08:52:22.874706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.874765 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:26.282592 (+3407827us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:26.282603 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.355995 (+ 73392us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.463568 (+107573us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.577139 (+113571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3407596}
I20251029 08:52:26.581144 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 931) took 3706 ms. Trace:
I20251029 08:52:26.581357 1179 rpcz_store.cc:276] 1029 08:52:22.874499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.874790 (+ 291us) service_pool.cc:225] Handling call
1029 08:52:26.580087 (+3705297us) ts_tablet_manager.cc:1151] Deleting tablet e2b87d133f8a4bd1877e07a1a8758147
1029 08:52:26.580983 (+ 896us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.581128 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704861}
I20251029 08:52:26.579052 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bbf6401423411089c9070f3b03f1a7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.579990 2603 tablet.cc:1722] T 713e102d9e1d40148636443a2260ccbc P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:26.583149 1974 ts_tablet_manager.cc:1929] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.583253 1974 log.cc:1199] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:26.583266 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 918) took 4014 ms. Trace:
I20251029 08:52:26.583604 1974 ts_tablet_manager.cc:1950] T eed66c3fa3f848df905a2a31fd553474 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:26.583927 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 969) took 3591 ms. Trace:
I20251029 08:52:26.584012 1974 rpcz_store.cc:276] 1029 08:52:22.992535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.992589 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:26.007939 (+3015350us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:26.007948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.417013 (+409065us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.583068 (+166055us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.583913 (+ 845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014446}
I20251029 08:52:26.584386 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 965) took 3663 ms. Trace:
I20251029 08:52:26.583664 1179 rpcz_store.cc:276] 1029 08:52:22.569120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.569173 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:26.582595 (+4013422us) ts_tablet_manager.cc:1151] Deleting tablet dbad8c4014b6423cb20e3e5c22ed69f1
1029 08:52:26.582722 (+ 127us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.583246 (+ 524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3703917}
I20251029 08:52:26.584455 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet eed66c3fa3f848df905a2a31fd553474 (table default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]) successfully deleted
I20251029 08:52:26.584457 1974 rpcz_store.cc:276] 1029 08:52:22.920570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.996414 (+ 75844us) service_pool.cc:225] Handling call
1029 08:52:26.584283 (+3587869us) ts_tablet_manager.cc:1151] Deleting tablet 61b901ffd7ae47f487a355f08c076a22
1029 08:52:26.584294 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.584372 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3587638}
I20251029 08:52:26.584770 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 973) took 3576 ms. Trace:
I20251029 08:52:26.584838 1974 rpcz_store.cc:276] 1029 08:52:23.008755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.008811 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:26.584666 (+3575855us) ts_tablet_manager.cc:1151] Deleting tablet 62597189ecb047788b83788b514e9099
1029 08:52:26.584684 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.584755 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3575587}
W20251029 08:52:26.584923 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 61b901ffd7ae47f487a355f08c076a22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b901ffd7ae47f487a355f08c076a22 already in progress: creating tablet
I20251029 08:52:26.585054 1974 tablet_replica.cc:333] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:26.585232 1974 raft_consensus.cc:2243] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:26.585350 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 62597189ecb047788b83788b514e9099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62597189ecb047788b83788b514e9099 already in progress: creating tablet
I20251029 08:52:26.585394 1974 raft_consensus.cc:2272] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.585569 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 933) took 3655 ms. Trace:
I20251029 08:52:26.585893 1974 ts_tablet_manager.cc:1916] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.586086 1177 tablet_bootstrap.cc:654] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.586084 1179 rpcz_store.cc:276] 1029 08:52:22.929597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.934024 (+ 4427us) service_pool.cc:225] Handling call
1029 08:52:26.584968 (+3650944us) ts_tablet_manager.cc:1151] Deleting tablet 61b901ffd7ae47f487a355f08c076a22
1029 08:52:26.584978 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.585433 (+ 455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3650670}
I20251029 08:52:26.587924 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 934) took 3647 ms. Trace:
I20251029 08:52:26.588514 1179 rpcz_store.cc:276] 1029 08:52:22.940859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.947683 (+ 6824us) service_pool.cc:225] Handling call
1029 08:52:26.587399 (+3639716us) ts_tablet_manager.cc:1151] Deleting tablet 6691e1944ae947b49e4431f31ba9ae41
1029 08:52:26.587509 (+ 110us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.587908 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3639414}
I20251029 08:52:26.590354 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aafa846cb2144cec9ce4b28e61d91961 (table default.Table_7c5175dfc80a4536bb7ff8968998073a [id=a7c4fc8558614b2e8bb6b6e4fdd6ca3e]) successfully deleted
I20251029 08:52:26.590487 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 935) took 3627 ms. Trace:
W20251029 08:52:26.591344 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e2b87d133f8a4bd1877e07a1a8758147 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2b87d133f8a4bd1877e07a1a8758147 already in progress: creating tablet
I20251029 08:52:26.591040 1179 rpcz_store.cc:276] 1029 08:52:22.963455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.964121 (+ 666us) service_pool.cc:225] Handling call
1029 08:52:26.589964 (+3625843us) ts_tablet_manager.cc:1151] Deleting tablet c18e0e01fdc54b8dab97ec339d2aaa07
1029 08:52:26.589973 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.590470 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3625452}
W20251029 08:52:26.591740 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet dbad8c4014b6423cb20e3e5c22ed69f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbad8c4014b6423cb20e3e5c22ed69f1 already in progress: creating tablet
I20251029 08:52:26.592469 1177 tablet_bootstrap.cc:492] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:26.592787 1177 ts_tablet_manager.cc:1403] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20251029 08:52:26.593946 1177 raft_consensus.cc:359] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.594154 1177 raft_consensus.cc:385] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.594238 1177 raft_consensus.cc:740] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:26.576471 720 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.592545 1179 tablet_replica.cc:333] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.594637 720 leader_election.cc:304] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.594630 1177 consensus_queue.cc:260] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.595213 1177 ts_tablet_manager.cc:1434] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20251029 08:52:26.595322 720 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.595450 720 leader_election.cc:304] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:26.595515 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 61b901ffd7ae47f487a355f08c076a22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b901ffd7ae47f487a355f08c076a22 already in progress: creating tablet
W20251029 08:52:26.595815 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6691e1944ae947b49e4431f31ba9ae41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6691e1944ae947b49e4431f31ba9ae41 already in progress: creating tablet
I20251029 08:52:26.595827 720 leader_election.cc:304] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.595634 1179 raft_consensus.cc:2243] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:26.596155 2636 raft_consensus.cc:2749] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.596311 2636 raft_consensus.cc:2749] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.596455 2636 raft_consensus.cc:2804] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:26.596235 996 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.596555 2636 raft_consensus.cc:697] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:26.596626 996 leader_election.cc:304] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.596772 2636 consensus_queue.cc:237] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.596946 2476 raft_consensus.cc:2749] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:26.597491 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c18e0e01fdc54b8dab97ec339d2aaa07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c18e0e01fdc54b8dab97ec339d2aaa07 already in progress: creating tablet
I20251029 08:52:26.599722 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb9c8c7286149128c2741635c242341 (DEFAULT_TABLE table=default.Table_096667619d1847a998aa237bb19f9024 [id=3bea892bcc4b444d868702d8156aebb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.599442 1179 raft_consensus.cc:2272] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.600122 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb9c8c7286149128c2741635c242341. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.603106 2451 raft_consensus.cc:493] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.603240 2451 raft_consensus.cc:515] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.603818 2451 leader_election.cc:290] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.603543 1179 ts_tablet_manager.cc:1916] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.604326 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba201a54466240129d82a096f9316242" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.604506 1068 raft_consensus.cc:2468] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:26.604892 861 leader_election.cc:304] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.605170 2451 raft_consensus.cc:2804] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.605239 2451 raft_consensus.cc:493] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.605309 2451 raft_consensus.cc:3060] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.608141 2451 raft_consensus.cc:515] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.609056 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba201a54466240129d82a096f9316242" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:26.609195 1068 raft_consensus.cc:3060] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.609256 2451 leader_election.cc:290] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.612010 1068 raft_consensus.cc:2468] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:26.612474 861 leader_election.cc:304] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.612732 2451 raft_consensus.cc:2804] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.612936 2451 raft_consensus.cc:697] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:26.613265 2451 consensus_queue.cc:237] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.620132 2548 raft_consensus.cc:493] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.620270 2548 raft_consensus.cc:515] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.620815 2548 leader_election.cc:290] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.621376 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa0ff14e19b4ad7a8b33796481f0f3f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.621518 1068 raft_consensus.cc:2468] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:26.621778 2359 raft_consensus.cc:493] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.621869 861 leader_election.cc:304] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.621906 2359 raft_consensus.cc:515] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.622130 2548 raft_consensus.cc:2804] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.622203 2548 raft_consensus.cc:493] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.622275 2548 raft_consensus.cc:3060] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.622789 2359 leader_election.cc:290] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.625197 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e213713757949caa3f504f70026b7b5" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.626286 2548 raft_consensus.cc:515] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.626974 2548 leader_election.cc:290] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.627486 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa0ff14e19b4ad7a8b33796481f0f3f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:26.627647 1068 raft_consensus.cc:3060] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:26.627463 996 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.630393 1068 raft_consensus.cc:2468] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:26.630868 998 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e213713757949caa3f504f70026b7b5
I20251029 08:52:26.630965 998 leader_election.cc:304] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.631438 861 leader_election.cc:304] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.631703 2548 raft_consensus.cc:2804] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.631911 2548 raft_consensus.cc:697] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:26.632086 2359 raft_consensus.cc:2749] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.632277 2548 consensus_queue.cc:237] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.632884 1183 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.633025 1183 ts_tablet_manager.cc:1403] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.000s
I20251029 08:52:26.633671 1183 raft_consensus.cc:359] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.634205 1183 raft_consensus.cc:385] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.634271 1183 raft_consensus.cc:740] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.634476 1183 consensus_queue.cc:260] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.635046 1183 ts_tablet_manager.cc:1434] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:26.651769 1179 ts_tablet_manager.cc:1929] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.654671 1179 log.cc:1199] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/eed66c3fa3f848df905a2a31fd553474
I20251029 08:52:26.658694 2554 raft_consensus.cc:493] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.658859 2554 raft_consensus.cc:515] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.659526 2554 leader_election.cc:290] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:26.660341 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa4f97075b3941aaa7cda3f9413e867f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.660542 1067 raft_consensus.cc:2468] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:26.660621 1179 ts_tablet_manager.cc:1950] T eed66c3fa3f848df905a2a31fd553474 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:26.661067 861 leader_election.cc:304] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.661862 2554 raft_consensus.cc:2804] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.661940 2554 raft_consensus.cc:493] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.662015 2554 raft_consensus.cc:3060] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.662921 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 937) took 3661 ms. Trace:
I20251029 08:52:26.663327 1178 raft_consensus.cc:740] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.663638 1178 consensus_queue.cc:260] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.664878 2554 raft_consensus.cc:515] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.665993 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa4f97075b3941aaa7cda3f9413e867f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:26.666142 1068 raft_consensus.cc:3060] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.663365 1179 rpcz_store.cc:276] 1029 08:52:23.000975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.001052 (+ 77us) service_pool.cc:225] Handling call
1029 08:52:26.592271 (+3591219us) ts_tablet_manager.cc:1151] Deleting tablet eed66c3fa3f848df905a2a31fd553474
1029 08:52:26.592410 (+ 139us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.625589 (+ 33179us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.650981 (+ 25392us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.662732 (+ 11751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590892}
I20251029 08:52:26.669498 1068 raft_consensus.cc:2468] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:26.669930 861 leader_election.cc:304] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:26.671936 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet eed66c3fa3f848df905a2a31fd553474 (table default.Table_efc1d3e55082423da1deead7cae760f4 [id=26ac01d14e174c19a7cc6aff086a4ec9]) successfully deleted
I20251029 08:52:26.672345 2554 leader_election.cc:290] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:26.672503 2554 raft_consensus.cc:2804] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.672595 2554 raft_consensus.cc:697] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:26.671456 1179 tablet_replica.cc:333] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.672837 2554 consensus_queue.cc:237] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.673847 1179 raft_consensus.cc:2243] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.674942 1179 raft_consensus.cc:2272] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.675509 2359 consensus_queue.cc:1048] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.679309 1179 ts_tablet_manager.cc:1916] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.681012 1183 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:26.682909 998 consensus_peers.cc:597] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6689cf0804f946c18ebc555a191549d4. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.684209 1183 tablet_bootstrap.cc:654] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:26.684499 720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 88701.8us
W20251029 08:52:26.684844 720 leader_election.cc:343] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.684955 720 leader_election.cc:304] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:26.685501 2458 consensus_queue.cc:1048] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.685894 2458 raft_consensus.cc:2749] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:26.696331 1178 ts_tablet_manager.cc:1434] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.149s user 0.001s sys 0.000s
I20251029 08:52:26.698123 1177 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:26.699956 1177 tablet_bootstrap.cc:654] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:26.708196 888 consensus_meta.cc:287] T f860052fec124192a67f28ffd43f9380 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent flushing consensus metadata: real 0.612s user 0.024s sys 0.000s
I20251029 08:52:26.718880 1183 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.719007 1183 ts_tablet_manager.cc:1403] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251029 08:52:26.721061 1183 raft_consensus.cc:359] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.721223 1183 raft_consensus.cc:385] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.721268 1183 raft_consensus.cc:740] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.721475 1183 consensus_queue.cc:260] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.721911 1183 ts_tablet_manager.cc:1434] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:26.726325 1179 ts_tablet_manager.cc:1929] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.726665 1974 ts_tablet_manager.cc:1929] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.726773 1974 log.cc:1199] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:26.727030 1178 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:26.727139 1974 ts_tablet_manager.cc:1950] T af38662420d14ac1a7cb5fd03fc1b9d8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:26.727412 1179 log.cc:1199] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:26.727895 1176 ts_tablet_manager.cc:1929] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:26.728008 1176 log.cc:1199] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:26.728317 1176 ts_tablet_manager.cc:1950] T bf29e030892d44e1ad4d1ca4f5aeab3b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:26.729174 1176 tablet_replica.cc:333] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.729295 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet bf29e030892d44e1ad4d1ca4f5aeab3b (table default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]) successfully deleted
I20251029 08:52:26.729339 1176 raft_consensus.cc:2243] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.729491 1176 raft_consensus.cc:2272] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.729971 1176 ts_tablet_manager.cc:1916] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:26.733446 861 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ba201a54466240129d82a096f9316242
W20251029 08:52:26.733803 861 leader_election.cc:343] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: ba201a54466240129d82a096f9316242
W20251029 08:52:26.734063 861 leader_election.cc:343] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4aa0ff14e19b4ad7a8b33796481f0f3f
W20251029 08:52:26.734297 861 leader_election.cc:343] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4aa0ff14e19b4ad7a8b33796481f0f3f
W20251029 08:52:26.734756 861 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa4f97075b3941aaa7cda3f9413e867f
W20251029 08:52:26.735013 861 consensus_peers.cc:597] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8c4d793f10147fba5e4f44474575bf0. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:26.735173 861 leader_election.cc:343] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:26.737438 1183 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.741057 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e522547c00dd4f70a3f03086eb6f372f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.741595 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e522547c00dd4f70a3f03086eb6f372f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.738030 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 04e64e5cf11547c1bfd6ac19b122efe4 (DEFAULT_TABLE table=default.Table_c204b45d751646c188724bd56add16c5 [id=06aad6e8b0ed4763857b806ff3aa4fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:26.744155 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e64e5cf11547c1bfd6ac19b122efe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:26.745108 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3421bab54d2445d09e0be9c5e15ac6da" } new_table_name: "default.Table_b60ea383469c4c7cb7e9783618b6a5ab" modify_external_catalogs: true
I20251029 08:52:26.746109 1179 ts_tablet_manager.cc:1950] T 1dd62833c9d548adabbbf71856a4b181 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:26.746448 2475 raft_consensus.cc:493] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.746601 2475 raft_consensus.cc:515] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.747331 2475 leader_election.cc:290] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.748070 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:26.748255 773 raft_consensus.cc:2468] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:26.748656 998 leader_election.cc:304] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:26.749291 2475 raft_consensus.cc:2804] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.749514 2554 consensus_queue.cc:1048] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:26.749387 2475 raft_consensus.cc:493] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.749794 2475 raft_consensus.cc:3060] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.752578 2475 raft_consensus.cc:515] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.753880 2475 leader_election.cc:290] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.749372 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 939) took 3748 ms. Trace:
W20251029 08:52:26.754645 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e27701fbf5b44e90ba33bed99fdc4d9b consensus state has no leader
I20251029 08:52:26.754927 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:26.755082 773 raft_consensus.cc:3060] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.757778 773 raft_consensus.cc:2468] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:26.757879 1183 tablet_bootstrap.cc:654] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.758311 998 leader_election.cc:304] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:26.758198 1179 rpcz_store.cc:276] 1029 08:52:23.001171 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.001234 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:26.669420 (+3668186us) ts_tablet_manager.cc:1151] Deleting tablet 1dd62833c9d548adabbbf71856a4b181
1029 08:52:26.669560 (+ 140us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.707986 (+ 38426us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.725927 (+ 17941us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.749351 (+ 23424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3667872}
I20251029 08:52:26.761831 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 1dd62833c9d548adabbbf71856a4b181 (table default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]) successfully deleted
I20251029 08:52:26.762076 2359 raft_consensus.cc:2804] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.762197 2359 raft_consensus.cc:697] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:26.762418 2359 consensus_queue.cc:237] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.762794 1179 tablet_replica.cc:333] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.765347 1179 raft_consensus.cc:2243] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:26.769807 1179 pending_rounds.cc:70] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:26.770323 1179 pending_rounds.cc:77] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037659462819840 op_type: NO_OP noop_request { }
I20251029 08:52:26.770572 1179 raft_consensus.cc:2889] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:26.770779 1179 raft_consensus.cc:2272] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.780107 1183 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.780443 1183 ts_tablet_manager.cc:1403] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.004s
I20251029 08:52:26.782177 1183 raft_consensus.cc:359] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.782598 1183 raft_consensus.cc:385] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.782793 1183 raft_consensus.cc:740] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.783233 1183 consensus_queue.cc:260] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.784337 1183 ts_tablet_manager.cc:1434] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251029 08:52:26.784969 1183 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.778832 1179 ts_tablet_manager.cc:1916] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.811407 1178 tablet_bootstrap.cc:654] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.816167 1183 tablet_bootstrap.cc:654] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.816275 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:26.816278 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fafe9a0731b48b3a60032d94623681d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.817842 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 987) took 3471 ms. Trace:
I20251029 08:52:26.817930 1974 rpcz_store.cc:276] 1029 08:52:23.346431 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.346486 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:26.585018 (+3238532us) ts_tablet_manager.cc:1151] Deleting tablet af38662420d14ac1a7cb5fd03fc1b9d8
1029 08:52:26.585026 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.722143 (+137117us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.726586 (+ 4443us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.817827 (+ 91241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3238322}
I20251029 08:52:26.818292 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 988) took 3132 ms. Trace:
I20251029 08:52:26.818347 1974 rpcz_store.cc:276] 1029 08:52:23.686144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.686464 (+ 320us) service_pool.cc:225] Handling call
1029 08:52:26.818199 (+3131735us) ts_tablet_manager.cc:1151] Deleting tablet 9493653c2e354458923e9e8dd0da0c60
1029 08:52:26.818208 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.818281 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3034505}
I20251029 08:52:26.818622 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 989) took 2921 ms. Trace:
I20251029 08:52:26.818676 1974 rpcz_store.cc:276] 1029 08:52:23.897380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.897693 (+ 313us) service_pool.cc:225] Handling call
1029 08:52:26.818539 (+2920846us) ts_tablet_manager.cc:1151] Deleting tablet d5bc2f2f5d6e418999fa40a6fadbbc71
1029 08:52:26.818546 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.818612 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920553}
I20251029 08:52:26.818866 1974 tablet_replica.cc:333] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:26.819026 1974 raft_consensus.cc:2243] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.819166 1974 raft_consensus.cc:2272] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.819695 1974 ts_tablet_manager.cc:1916] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.820467 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet af38662420d14ac1a7cb5fd03fc1b9d8 (table default.Table_69a638baaadc493f9439102862bb7607 [id=14f726f4d8104eaf99f5dbb58f9ce22c]) successfully deleted
W20251029 08:52:26.820737 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9493653c2e354458923e9e8dd0da0c60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9493653c2e354458923e9e8dd0da0c60 already in progress: creating tablet
W20251029 08:52:26.820906 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bc2f2f5d6e418999fa40a6fadbbc71 already in progress: creating tablet
W20251029 08:52:26.816792 996 consensus_peers.cc:597] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:26.827432 996 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:26.827875 996 leader_election.cc:343] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:26.844069 2554 raft_consensus.cc:493] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.844273 2554 raft_consensus.cc:515] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.845060 2554 leader_election.cc:290] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:26.846733 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c1a8a1e4cc14f1aa6e4f91174f45127" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:26.846976 1067 raft_consensus.cc:2468] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:26.847608 861 leader_election.cc:304] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:26.848300 861 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:26.848825 2554 raft_consensus.cc:2804] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.848930 2554 raft_consensus.cc:493] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.849022 2554 raft_consensus.cc:3060] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.849829 1183 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.849960 1183 ts_tablet_manager.cc:1403] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.002s
I20251029 08:52:26.850713 1183 raft_consensus.cc:359] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.850869 1183 raft_consensus.cc:385] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.850914 1183 raft_consensus.cc:740] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.851102 1183 consensus_queue.cc:260] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.851604 1183 ts_tablet_manager.cc:1434] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:26.851811 1974 ts_tablet_manager.cc:1929] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.851814 2359 raft_consensus.cc:493] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.851902 1183 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.851905 1974 log.cc:1199] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7b5051ff776e4975b917db0393a1f945
I20251029 08:52:26.851939 2359 raft_consensus.cc:515] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.852293 1974 ts_tablet_manager.cc:1950] T 7b5051ff776e4975b917db0393a1f945 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:26.852627 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 990) took 2955 ms. Trace:
I20251029 08:52:26.852742 2359 leader_election.cc:290] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.852702 1974 rpcz_store.cc:276] 1029 08:52:23.897524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.898046 (+ 522us) service_pool.cc:225] Handling call
1029 08:52:26.818830 (+2920784us) ts_tablet_manager.cc:1151] Deleting tablet 7b5051ff776e4975b917db0393a1f945
1029 08:52:26.818837 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.848535 (+ 29698us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.851706 (+ 3171us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.852602 (+ 896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920647}
I20251029 08:52:26.853194 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7b5051ff776e4975b917db0393a1f945 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:26.853358 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0ef904c2fb4dfeadbf56ae659c4e20" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.853561 924 raft_consensus.cc:2468] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:26.854115 996 leader_election.cc:304] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:26.855335 998 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:26.855942 2359 raft_consensus.cc:2804] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:26.856047 2359 raft_consensus.cc:493] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:26.856118 2359 raft_consensus.cc:3060] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.862448 1177 tablet_bootstrap.cc:492] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:26.862661 1177 ts_tablet_manager.cc:1403] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.165s user 0.007s sys 0.005s
I20251029 08:52:26.863481 1177 raft_consensus.cc:359] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.863713 1177 raft_consensus.cc:385] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.863812 1177 raft_consensus.cc:740] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.864090 1177 consensus_queue.cc:260] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.864714 1177 ts_tablet_manager.cc:1434] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:26.865001 1177 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:26.864926 2359 raft_consensus.cc:515] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.866329 1177 tablet_bootstrap.cc:654] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.870014 1179 ts_tablet_manager.cc:1929] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.870128 1179 log.cc:1199] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/24ab9df6f768410eb4f66cffdcd68df2
I20251029 08:52:26.870472 1179 ts_tablet_manager.cc:1950] T 24ab9df6f768410eb4f66cffdcd68df2 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:26.870782 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 938) took 3869 ms. Trace:
I20251029 08:52:26.870857 1179 rpcz_store.cc:276] 1029 08:52:23.001075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.001329 (+ 254us) service_pool.cc:225] Handling call
1029 08:52:26.762228 (+3760899us) ts_tablet_manager.cc:1151] Deleting tablet 24ab9df6f768410eb4f66cffdcd68df2
1029 08:52:26.762243 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.821767 (+ 59524us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.869901 (+ 48134us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.870765 (+ 864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3760335}
I20251029 08:52:26.871212 1179 tablet_replica.cc:333] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.871397 1179 raft_consensus.cc:2243] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:26.871462 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 24ab9df6f768410eb4f66cffdcd68df2 (table default.Table_3fffb85f46a84893b72083c6894e6f84 [id=c353c4db7cf64e628d0049f67aae8cca]) successfully deleted
I20251029 08:52:26.871546 1179 raft_consensus.cc:2272] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:26.871785 998 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:26.871922 2359 leader_election.cc:290] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.874099 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_70f74ef51bd84663a6a30cbd00c63bb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.874666 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70f74ef51bd84663a6a30cbd00c63bb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.876977 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7a2164d9430240f8a00e62963f6edb46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.877475 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a2164d9430240f8a00e62963f6edb46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.878885 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_533c60a10a78409e9fd7e95fd47ecdd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.879346 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_533c60a10a78409e9fd7e95fd47ecdd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.891517 1179 ts_tablet_manager.cc:1916] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.892406 1183 tablet_bootstrap.cc:654] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:26.911823 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 900e8494364247289fa4208d81dfe3bb consensus state has no leader
W20251029 08:52:26.912541 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d24187f389a45628d21370c5b29a537 consensus state has no leader
W20251029 08:52:26.868181 861 consensus_peers.cc:597] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b07eb4befa41f3a1b5f2e9ebc2277c. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:26.932142 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 63992.4us
I20251029 08:52:26.932375 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66cf9e18bddd418e88537d60dbac69ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:26.870354 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0ef904c2fb4dfeadbf56ae659c4e20" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:26.932888 924 raft_consensus.cc:3060] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:26.932948 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66cf9e18bddd418e88537d60dbac69ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.934381 1183 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:26.934365 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02" } modify_external_catalogs: true
I20251029 08:52:26.934548 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d0f2869dc6b94cadbeeba0c8f4b61e02" } modify_external_catalogs: true
I20251029 08:52:26.934588 1183 ts_tablet_manager.cc:1403] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.000s
I20251029 08:52:26.935122 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c068416bd1584f84942bc94450654170"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.935586 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c068416bd1584f84942bc94450654170 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:26.935726 924 raft_consensus.cc:2468] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:26.872133 2554 raft_consensus.cc:515] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.936946 2554 leader_election.cc:290] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:26.938023 996 leader_election.cc:304] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:26.938491 2485 raft_consensus.cc:2804] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.938612 2485 raft_consensus.cc:697] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:26.938850 2485 consensus_queue.cc:237] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.942580 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c1a8a1e4cc14f1aa6e4f91174f45127" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:26.942764 1068 raft_consensus.cc:3060] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:26.942858 1183 raft_consensus.cc:359] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:26.944898 861 leader_election.cc:343] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3c1a8a1e4cc14f1aa6e4f91174f45127
W20251029 08:52:26.945432 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 165d46a006c445b7bf500de4f0a8ba70 consensus state has no leader
I20251029 08:52:26.945673 1068 raft_consensus.cc:2468] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:26.946128 861 leader_election.cc:304] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:26.946611 2295 raft_consensus.cc:2804] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:26.946277 1183 raft_consensus.cc:385] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.946741 2295 raft_consensus.cc:697] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:26.946960 2295 consensus_queue.cc:237] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:26.946801 1183 raft_consensus.cc:740] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.948091 1183 consensus_queue.cc:260] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.956342 1183 ts_tablet_manager.cc:1434] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20251029 08:52:26.956938 1179 ts_tablet_manager.cc:1929] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:26.958031 1179 log.cc:1199] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:26.961007 1183 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:26.963160 1178 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:26.963384 1178 ts_tablet_manager.cc:1403] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.236s user 0.004s sys 0.000s
I20251029 08:52:26.963554 1179 ts_tablet_manager.cc:1950] T a2b7853b85d24a04b8b11caaa4fe8c4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:26.900578 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:26.966617 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7fd53f3b59d14ad39715bc62e513ab79" } modify_external_catalogs: true
I20251029 08:52:26.966854 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7fd53f3b59d14ad39715bc62e513ab79" } modify_external_catalogs: true
W20251029 08:52:26.967047 996 consensus_peers.cc:597] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:26.967651 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_95ce9269797c4de3b5bd2c1cc2a6f384" } modify_external_catalogs: true
I20251029 08:52:26.967792 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_95ce9269797c4de3b5bd2c1cc2a6f384" } modify_external_catalogs: true
I20251029 08:52:26.968326 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57929518185c4558a0f9f12c40945be2" } modify_external_catalogs: true
I20251029 08:52:26.968449 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_57929518185c4558a0f9f12c40945be2" } modify_external_catalogs: true
I20251029 08:52:26.969508 1178 raft_consensus.cc:359] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.969771 1178 raft_consensus.cc:385] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:26.969885 1178 raft_consensus.cc:740] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:26.970345 1176 ts_tablet_manager.cc:1929] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:26.970481 1176 log.cc:1199] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e3c24c0d32f74435bd9d28f2415e2c47
I20251029 08:52:26.970803 1176 ts_tablet_manager.cc:1950] T e3c24c0d32f74435bd9d28f2415e2c47 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:26.971904 1176 tablet_replica.cc:333] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:26.971889 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e3c24c0d32f74435bd9d28f2415e2c47 (table default.Table_3222378209e141649a30f5c940ae2c23 [id=b189fe1c508d4520a69d483cb97d4886]) successfully deleted
W20251029 08:52:26.972417 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a73a891fb6034f989d29a3f837303d84 because the tablet was not found. No further retry: Not found: Tablet not found: a73a891fb6034f989d29a3f837303d84
I20251029 08:52:26.971978 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 940) took 3970 ms. Trace:
I20251029 08:52:26.931622 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 991) took 3034 ms. Trace:
I20251029 08:52:26.972595 1179 rpcz_store.cc:276] 1029 08:52:23.001330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.001540 (+ 210us) service_pool.cc:225] Handling call
1029 08:52:26.871164 (+3869624us) ts_tablet_manager.cc:1151] Deleting tablet a2b7853b85d24a04b8b11caaa4fe8c4d
1029 08:52:26.871175 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.921559 (+ 50384us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.955558 (+ 33999us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:26.971737 (+ 16179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41344,"tablet-delete.queue_time_us":3869470}
I20251029 08:52:26.974267 1974 rpcz_store.cc:276] 1029 08:52:23.897624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.898219 (+ 595us) service_pool.cc:225] Handling call
1029 08:52:26.931451 (+3033232us) ts_tablet_manager.cc:1151] Deleting tablet a92cc4a93a584dffaafe21316fe8dc06
1029 08:52:26.931467 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.931596 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3033016}
I20251029 08:52:26.974602 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a2b7853b85d24a04b8b11caaa4fe8c4d (table default.Table_10573b6a1c074383b5808291ca1505ed [id=38c2c6eda2954cd3a5640c08ad4ce417]) successfully deleted
W20251029 08:52:26.974927 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a92cc4a93a584dffaafe21316fe8dc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92cc4a93a584dffaafe21316fe8dc06 already in progress: creating tablet
I20251029 08:52:26.975322 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 993) took 3064 ms. Trace:
I20251029 08:52:26.975402 1974 rpcz_store.cc:276] 1029 08:52:23.910548 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.910604 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:26.975173 (+3064569us) ts_tablet_manager.cc:1151] Deleting tablet fb22af7515084e099b41f9c1c92c067a
1029 08:52:26.975185 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.975303 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064369}
W20251029 08:52:26.975781 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet fb22af7515084e099b41f9c1c92c067a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb22af7515084e099b41f9c1c92c067a already in progress: creating tablet
I20251029 08:52:26.975958 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 941) took 3965 ms. Trace:
I20251029 08:52:26.970280 1178 consensus_queue.cc:260] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.977476 1183 tablet_bootstrap.cc:654] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.977953 1178 ts_tablet_manager.cc:1434] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251029 08:52:26.976724 1179 rpcz_store.cc:276] 1029 08:52:23.010656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.010748 (+ 92us) service_pool.cc:225] Handling call
1029 08:52:26.974916 (+3964168us) ts_tablet_manager.cc:1151] Deleting tablet 62597189ecb047788b83788b514e9099
1029 08:52:26.975147 (+ 231us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.975934 (+ 787us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3963940}
I20251029 08:52:26.978220 1178 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:26.978278 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 62597189ecb047788b83788b514e9099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62597189ecb047788b83788b514e9099 already in progress: creating tablet
I20251029 08:52:26.979427 1179 tablet_replica.cc:333] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:26.980319 1178 tablet_bootstrap.cc:654] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:26.980713 1179 raft_consensus.cc:2243] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:26.983899 1974 tablet_replica.cc:333] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:26.984431 1974 raft_consensus.cc:2243] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:26.984177 1179 raft_consensus.cc:2272] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.984849 1974 raft_consensus.cc:2272] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:26.985471 1974 ts_tablet_manager.cc:1916] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.987279 1179 ts_tablet_manager.cc:1916] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:26.996346 2475 raft_consensus.cc:493] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:26.996511 2475 raft_consensus.cc:515] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:26.997148 2475 leader_election.cc:290] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:26.998216 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0a016861cf8d4f54a567239750fc81df" } modify_external_catalogs: true
I20251029 08:52:26.998404 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0a016861cf8d4f54a567239750fc81df" } modify_external_catalogs: true
I20251029 08:52:26.998641 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf29e030892d44e1ad4d1ca4f5aeab3b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:26.999704 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b2c950d7747b4bfebfb385e8fb2f8d59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:26.999825 996 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:27.000324 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2c950d7747b4bfebfb385e8fb2f8d59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.000804 998 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:27.000932 998 leader_election.cc:304] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.001446 2475 raft_consensus.cc:2749] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.011804 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_438d2aad28fd458786870f8dd77671ec" } new_table_name: "default.Table_ca344fa2220d4ea09a867ad42d7fe215" modify_external_catalogs: true
I20251029 08:52:27.015046 590 catalog_manager.cc:5649] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:27.015445 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:27.015703 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]). Expected version 1 got 0
I20251029 08:52:27.016116 745 tablet_service.cc:1505] Processing CreateTablet for tablet f27e7f597b20453286592dfd3149ff50 (DEFAULT_TABLE table=default.Table_5a7aba9b7fad44f899406973fdc3a841 [id=265ece7684304b329f9db5cb744fed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.016613 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e7f597b20453286592dfd3149ff50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.017822 1045 tablet_service.cc:1505] Processing CreateTablet for tablet f27e7f597b20453286592dfd3149ff50 (DEFAULT_TABLE table=default.Table_5a7aba9b7fad44f899406973fdc3a841 [id=265ece7684304b329f9db5cb744fed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.018239 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e7f597b20453286592dfd3149ff50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.018319 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_ba1277f42faa48bf9e570e5ad794055b [id=3daf859c83c740d7a701a8bbd8aa2ab8] alter complete (version 3)
I20251029 08:52:27.021005 758 tablet_service.cc:1505] Processing CreateTablet for tablet 532325e90d2546dd9d3412002651fbe9 (DEFAULT_TABLE table=default.Table_ebb42f8e232e40eca333dd1e8b60cc62 [id=8b85cf05aba441058ed19fd4e4cf57f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.021435 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532325e90d2546dd9d3412002651fbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.025339 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 532325e90d2546dd9d3412002651fbe9 (DEFAULT_TABLE table=default.Table_ebb42f8e232e40eca333dd1e8b60cc62 [id=8b85cf05aba441058ed19fd4e4cf57f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.025799 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532325e90d2546dd9d3412002651fbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.025966 757 tablet_service.cc:1505] Processing CreateTablet for tablet c5062ddafd784a428158aa537955eddd (DEFAULT_TABLE table=default.Table_eba74fd181d54950825081b6abd3201a [id=fb690a674cf840e0a3dd7f284480f75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.026003 893 tablet_service.cc:1505] Processing CreateTablet for tablet 532325e90d2546dd9d3412002651fbe9 (DEFAULT_TABLE table=default.Table_ebb42f8e232e40eca333dd1e8b60cc62 [id=8b85cf05aba441058ed19fd4e4cf57f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.026314 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5062ddafd784a428158aa537955eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.026407 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532325e90d2546dd9d3412002651fbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.028282 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c5062ddafd784a428158aa537955eddd (DEFAULT_TABLE table=default.Table_eba74fd181d54950825081b6abd3201a [id=fb690a674cf840e0a3dd7f284480f75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.028657 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5062ddafd784a428158aa537955eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.028817 897 tablet_service.cc:1505] Processing CreateTablet for tablet c5062ddafd784a428158aa537955eddd (DEFAULT_TABLE table=default.Table_eba74fd181d54950825081b6abd3201a [id=fb690a674cf840e0a3dd7f284480f75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.029160 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5062ddafd784a428158aa537955eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.037739 888 tablet_service.cc:1505] Processing CreateTablet for tablet f27e7f597b20453286592dfd3149ff50 (DEFAULT_TABLE table=default.Table_5a7aba9b7fad44f899406973fdc3a841 [id=265ece7684304b329f9db5cb744fed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.038146 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e7f597b20453286592dfd3149ff50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.040390 896 tablet_service.cc:1505] Processing CreateTablet for tablet c7644fbf3d974c299f1ed550ede477f8 (DEFAULT_TABLE table=default.Table_4fa1c998b783428e81198c6c52b5f905 [id=a8c5d91aca7d43ba815a3c05b3003be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.041016 591 master_service.cc:954] ReplaceTablet: received request to replace tablet 2fad7549758649229ad771de926aaf15 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:27.041122 757 tablet_service.cc:1505] Processing CreateTablet for tablet c7644fbf3d974c299f1ed550ede477f8 (DEFAULT_TABLE table=default.Table_4fa1c998b783428e81198c6c52b5f905 [id=a8c5d91aca7d43ba815a3c05b3003be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.041536 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7644fbf3d974c299f1ed550ede477f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.042752 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7644fbf3d974c299f1ed550ede477f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.044925 749 tablet_service.cc:1505] Processing CreateTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 (DEFAULT_TABLE table=default.Table_13d40f6bbc974e0bb9e249282fe3a279 [id=229296af63d9476e8fba89f61e81e848]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.045245 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d2df9e1e744ed1a6cf549ab57ed266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.047642 1178 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.047781 1178 ts_tablet_manager.cc:1403] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251029 08:52:27.048434 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8c3da4b29e3c4208ae2109789f994c1e
I20251029 08:52:27.048570 1178 raft_consensus.cc:359] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.048730 1178 raft_consensus.cc:385] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.049168 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3efff2709af84a29acfd361aa9213152"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.049551 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:27.049657 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3efff2709af84a29acfd361aa9213152 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.049774 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.050204 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3da4b29e3c4208ae2109789f994c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.051874 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c7644fbf3d974c299f1ed550ede477f8 (DEFAULT_TABLE table=default.Table_4fa1c998b783428e81198c6c52b5f905 [id=a8c5d91aca7d43ba815a3c05b3003be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.052304 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7644fbf3d974c299f1ed550ede477f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.052371 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07bf5d693be240f18bacccc81a6ea5d4 consensus state has no leader
I20251029 08:52:27.048782 1178 raft_consensus.cc:740] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.052932 1178 consensus_queue.cc:260] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.056766 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d965d5a6b1324b00ad68b28d76796e80
I20251029 08:52:27.056915 886 tablet_service.cc:1505] Processing CreateTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 (DEFAULT_TABLE table=default.Table_13d40f6bbc974e0bb9e249282fe3a279 [id=229296af63d9476e8fba89f61e81e848]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.057312 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d2df9e1e744ed1a6cf549ab57ed266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.057355 905 tablet_service.cc:1552] Processing DeleteTablet for tablet d965d5a6b1324b00ad68b28d76796e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.057746 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet d965d5a6b1324b00ad68b28d76796e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.057883 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet af913080bfe44a86a76704ffa32cd9e1
I20251029 08:52:27.058063 1044 tablet_service.cc:1505] Processing CreateTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 (DEFAULT_TABLE table=default.Table_13d40f6bbc974e0bb9e249282fe3a279 [id=229296af63d9476e8fba89f61e81e848]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.058398 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d2df9e1e744ed1a6cf549ab57ed266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.058463 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 47f58be6d7f5492395ca6192f7ce0930 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:27.058643 747 tablet_service.cc:1552] Processing DeleteTablet for tablet af913080bfe44a86a76704ffa32cd9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.058737 745 tablet_service.cc:1505] Processing CreateTablet for tablet 30a9b421e55b43d790395d78d528e8db (DEFAULT_TABLE table=default.Table_d8b2eaecf62d466a9bfe3a63d1b9d838 [id=c645f56123cf4a4081bf1c8ed503c045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.058894 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_5471f52640fd473e90327f739bd8223d [id=559cd9144082414dade2ac5d73a7057c] alter complete (version 2)
I20251029 08:52:27.058993 905 tablet_service.cc:1552] Processing DeleteTablet for tablet af913080bfe44a86a76704ffa32cd9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.059058 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a9b421e55b43d790395d78d528e8db. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.059785 596 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:52653 (request call id 179) took 943 ms (client timeout 1000 ms). Trace:
I20251029 08:52:27.060001 905 tablet_service.cc:1505] Processing CreateTablet for tablet 30a9b421e55b43d790395d78d528e8db (DEFAULT_TABLE table=default.Table_d8b2eaecf62d466a9bfe3a63d1b9d838 [id=c645f56123cf4a4081bf1c8ed503c045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.060359 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a9b421e55b43d790395d78d528e8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.060345 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 58932f38c0d64aec9a02238eaea56a25
W20251029 08:52:27.063542 595 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58367 (request call id 157) took 939 ms (client timeout 1000 ms). Trace:
W20251029 08:52:27.059870 596 rpcz_store.cc:269] 1029 08:52:26.116587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.126564 (+ 9977us) service_pool.cc:225] Handling call
1029 08:52:27.059766 (+933202us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:26.243181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:26.246521 (+ 3340us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:26.246914 (+ 393us) write_op.cc:435] Acquired schema lock
1029 08:52:26.246917 (+ 3us) tablet.cc:660] Decoding operations
1029 08:52:26.248619 (+ 1702us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:26.248636 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:26.248645 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
1029 08:52:26.249975 (+ 1330us) tablet.cc:699] Row locks acquired
1029 08:52:26.249978 (+ 3us) write_op.cc:260] PREPARE: finished
1029 08:52:26.250837 (+ 859us) write_op.cc:270] Start()
1029 08:52:26.251741 (+ 904us) write_op.cc:276] Timestamp: P: 1761727946250355 usec, L: 0
1029 08:52:26.251746 (+ 5us) op_driver.cc:348] REPLICATION: starting
1029 08:52:26.254833 (+ 3087us) log.cc:844] Serialized 356 byte log entry
1029 08:52:26.260476 (+ 5643us) op_driver.cc:464] REPLICATION: finished
1029 08:52:26.260948 (+ 472us) write_op.cc:301] APPLY: starting
1029 08:52:26.260966 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:26.261004 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:26.261029 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:26.261073 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:26.261110 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:26.261137 (+ 27us) write_op.cc:312] APPLY: finished
1029 08:52:26.261223 (+ 86us) log.cc:844] Serialized 23 byte log entry
1029 08:52:26.261279 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:26.261299 (+ 20us) write_op.cc:454] Released schema lock
1029 08:52:26.261305 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:26.261372 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:26.261378 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
1029 08:52:27.035843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:27.035926 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:27.035932 (+ 6us) write_op.cc:435] Acquired schema lock
1029 08:52:27.035934 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:27.036002 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:27.036012 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:27.036043 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
1029 08:52:27.036089 (+ 46us) tablet.cc:699] Row locks acquired
1029 08:52:27.036091 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:27.036116 (+ 25us) write_op.cc:270] Start()
1029 08:52:27.036138 (+ 22us) write_op.cc:276] Timestamp: P: 1761727947036085 usec, L: 0
1029 08:52:27.036141 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 08:52:27.036270 (+ 129us) log.cc:844] Serialized 356 byte log entry
1029 08:52:27.052772 (+ 16502us) op_driver.cc:464] REPLICATION: finished
1029 08:52:27.055653 (+ 2881us) write_op.cc:301] APPLY: starting
1029 08:52:27.055663 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:27.055689 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:27.055722 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:27.055757 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:27.055781 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:27.055787 (+ 6us) write_op.cc:312] APPLY: finished
1029 08:52:27.055845 (+ 58us) log.cc:844] Serialized 23 byte log entry
1029 08:52:27.055881 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:27.055891 (+ 10us) write_op.cc:454] Released schema lock
1029 08:52:27.055896 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:27.055954 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:27.055958 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":411,"apply.run_cpu_time_us":420,"apply.run_wall_time_us":421,"num_ops":1,"prepare.queue_time_us":113057,"prepare.run_cpu_time_us":1149,"prepare.run_wall_time_us":14239,"raft.queue_time_us":525,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":198,"replication_time_us":8700}],["op",{"apply.queue_time_us":2842,"apply.run_cpu_time_us":302,"apply.run_wall_time_us":301,"num_ops":1,"prepare.queue_time_us":16877,"prepare.run_cpu_time_us":472,"prepare.run_wall_time_us":471,"replication_time_us":16615}]]}
I20251029 08:52:27.065207 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa0ff14e19b4ad7a8b33796481f0f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.065263 899 tablet_service.cc:1552] Processing DeleteTablet for tablet ba201a54466240129d82a096f9316242 with delete_type TABLET_DATA_DELETED (replaced by tablet cf09e8fd3d2b43318fa37d922c8b826f) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.065348 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 58932f38c0d64aec9a02238eaea56a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.067584 2475 consensus_queue.cc:1048] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:27.069312 749 tablet_service.cc:1552] Processing DeleteTablet for tablet d965d5a6b1324b00ad68b28d76796e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.069489 2475 raft_consensus.cc:493] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.069620 2475 raft_consensus.cc:515] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.070204 2475 leader_election.cc:290] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.070746 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3955ff79faf42bb82f764aae79b85d9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:27.071352 996 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.071692 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.071887 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 58932f38c0d64aec9a02238eaea56a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.072046 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.072233 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 30a9b421e55b43d790395d78d528e8db (DEFAULT_TABLE table=default.Table_d8b2eaecf62d466a9bfe3a63d1b9d838 [id=c645f56123cf4a4081bf1c8ed503c045]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.072386 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 58932f38c0d64aec9a02238eaea56a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.072664 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a9b421e55b43d790395d78d528e8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.072868 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee83f5526f74ba2a344edd97db94050 (DEFAULT_TABLE table=default.Table_ac500003d4a84f739ff1c0ac8d40e93c [id=ff2c0066d55c4ce5a52c1def8963d70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.073205 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee83f5526f74ba2a344edd97db94050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.073926 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet af913080bfe44a86a76704ffa32cd9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.074348 905 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb1f65610f446b3ba6da5b52e55f156 (DEFAULT_TABLE table=default.Table_d2bfdd8e87004d21a2742e8b4780a151 [id=153ba6a1569d4089a44a4865b93a28cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.074705 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb1f65610f446b3ba6da5b52e55f156. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.076352 998 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:27.076457 998 leader_election.cc:304] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.076476 750 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb1f65610f446b3ba6da5b52e55f156 (DEFAULT_TABLE table=default.Table_d2bfdd8e87004d21a2742e8b4780a151 [id=153ba6a1569d4089a44a4865b93a28cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.076779 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb1f65610f446b3ba6da5b52e55f156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.076848 2475 raft_consensus.cc:2749] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.077131 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb1f65610f446b3ba6da5b52e55f156 (DEFAULT_TABLE table=default.Table_d2bfdd8e87004d21a2742e8b4780a151 [id=153ba6a1569d4089a44a4865b93a28cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.077446 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb1f65610f446b3ba6da5b52e55f156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.078639 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee83f5526f74ba2a344edd97db94050 (DEFAULT_TABLE table=default.Table_ac500003d4a84f739ff1c0ac8d40e93c [id=ff2c0066d55c4ce5a52c1def8963d70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.078974 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee83f5526f74ba2a344edd97db94050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.083833 1178 ts_tablet_manager.cc:1434] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251029 08:52:27.084228 1178 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.086164 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.086795 1178 tablet_bootstrap.cc:654] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.094595 1177 tablet_bootstrap.cc:492] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:27.094846 1177 ts_tablet_manager.cc:1403] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.230s user 0.002s sys 0.004s
I20251029 08:52:27.095640 1177 raft_consensus.cc:359] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.095908 1177 raft_consensus.cc:385] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.096120 1177 raft_consensus.cc:740] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.096575 1177 consensus_queue.cc:260] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:27.099066 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2f816afb0d084c6db44be8a41f53ce04 (table default.Table_21d20ee24eca44568b6a69044d5c55a4 [id=436174f0eefd441aa0719ec72739aba6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2f816afb0d084c6db44be8a41f53ce04 consensus state has no leader
W20251029 08:52:27.063619 595 rpcz_store.cc:269] 1029 08:52:26.124083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.178041 (+ 53958us) service_pool.cc:225] Handling call
1029 08:52:27.063523 (+885482us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:26.741974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:26.742102 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:26.742107 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:26.742109 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:26.742235 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:26.742242 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:26.742271 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
1029 08:52:26.742409 (+ 138us) tablet.cc:699] Row locks acquired
1029 08:52:26.742411 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:26.742437 (+ 26us) write_op.cc:270] Start()
1029 08:52:26.742561 (+ 124us) write_op.cc:276] Timestamp: P: 1761727946742383 usec, L: 0
1029 08:52:26.742565 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 08:52:26.743628 (+ 1063us) log.cc:844] Serialized 527 byte log entry
1029 08:52:26.869837 (+126209us) op_driver.cc:464] REPLICATION: finished
1029 08:52:26.870600 (+ 763us) write_op.cc:301] APPLY: starting
1029 08:52:26.870613 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:26.870650 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:26.870685 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:26.870725 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:26.870750 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:26.870756 (+ 6us) write_op.cc:312] APPLY: finished
1029 08:52:26.870811 (+ 55us) log.cc:844] Serialized 23 byte log entry
1029 08:52:26.870851 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:26.870864 (+ 13us) write_op.cc:454] Released schema lock
1029 08:52:26.870870 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":722,"apply.run_cpu_time_us":326,"apply.run_wall_time_us":326,"num_ops":1,"prepare.queue_time_us":484354,"prepare.run_cpu_time_us":9337,"prepare.run_wall_time_us":125624,"raft.queue_time_us":755,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":127251,"thread_start_us":122579,"threads_started":1}]]}
I20251029 08:52:27.103742 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 646aaf0453134f3c9306a68b45d1fa64 (DEFAULT_TABLE table=default.Table_01d382774b104ee89690fbab1a407cab [id=711d991156fb4ffbbe95c68731f50de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.104169 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646aaf0453134f3c9306a68b45d1fa64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.114006 745 tablet_service.cc:1505] Processing CreateTablet for tablet 646aaf0453134f3c9306a68b45d1fa64 (DEFAULT_TABLE table=default.Table_01d382774b104ee89690fbab1a407cab [id=711d991156fb4ffbbe95c68731f50de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.119247 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646aaf0453134f3c9306a68b45d1fa64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.120205 1176 raft_consensus.cc:2243] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:27.120502 1176 pending_rounds.cc:70] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Trying to abort 1 pending ops.
I20251029 08:52:27.120570 1176 pending_rounds.cc:77] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037669262430208 op_type: NO_OP noop_request { }
I20251029 08:52:27.120703 1176 raft_consensus.cc:2889] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:27.120793 1176 raft_consensus.cc:2272] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.122573 1176 ts_tablet_manager.cc:1916] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.124833 2458 raft_consensus.cc:493] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.124907 1183 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.125039 1183 ts_tablet_manager.cc:1403] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.165s user 0.005s sys 0.003s
I20251029 08:52:27.124977 2458 raft_consensus.cc:515] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.125623 2458 leader_election.cc:290] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.125860 1183 raft_consensus.cc:359] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.126338 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf9e36c47be4509a998be8e7138475b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.126608 1183 raft_consensus.cc:385] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.126689 1183 raft_consensus.cc:740] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.126817 758 tablet_service.cc:1505] Processing CreateTablet for tablet 67c5cd2e25e34da9870af4a1edbcf91f (DEFAULT_TABLE table=default.Table_3dbf386f200442c097b8998936081933 [id=60a1c9468a4e4642816ffcc7c52a0548]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.126870 720 leader_election.cc:343] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.127039 1183 consensus_queue.cc:260] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.127286 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c5cd2e25e34da9870af4a1edbcf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.127596 1183 ts_tablet_manager.cc:1434] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:27.128777 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:27.133183 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 67c5cd2e25e34da9870af4a1edbcf91f (DEFAULT_TABLE table=default.Table_3dbf386f200442c097b8998936081933 [id=60a1c9468a4e4642816ffcc7c52a0548]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.137049 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c5cd2e25e34da9870af4a1edbcf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.137169 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3498dba075944290bd5ea3c68f6578eb (DEFAULT_TABLE table=default.Table_064cacab68c74e94a8224b2c068aec83 [id=69c514478c9c4cedbe4be3d65e769616]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.137603 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3498dba075944290bd5ea3c68f6578eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.098590 886 tablet_service.cc:1505] Processing CreateTablet for tablet 646aaf0453134f3c9306a68b45d1fa64 (DEFAULT_TABLE table=default.Table_01d382774b104ee89690fbab1a407cab [id=711d991156fb4ffbbe95c68731f50de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.100150 899 tablet_service.cc:1505] Processing CreateTablet for tablet 67c5cd2e25e34da9870af4a1edbcf91f (DEFAULT_TABLE table=default.Table_3dbf386f200442c097b8998936081933 [id=60a1c9468a4e4642816ffcc7c52a0548]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.144543 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c5cd2e25e34da9870af4a1edbcf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.145226 1178 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.145432 1178 ts_tablet_manager.cc:1403] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20251029 08:52:27.145238 1177 ts_tablet_manager.cc:1434] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20251029 08:52:27.145586 1183 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.145993 1177 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:27.146243 1178 raft_consensus.cc:359] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.146576 1178 raft_consensus.cc:385] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.146708 1178 raft_consensus.cc:740] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.147064 1178 consensus_queue.cc:260] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.147889 1183 tablet_bootstrap.cc:654] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.149747 1178 ts_tablet_manager.cc:1434] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 08:52:27.150080 1178 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.150974 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bd357a723a1d4bc1aa4b480c8a29ef03
I20251029 08:52:27.151715 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d089089a84af48068f5a61c67b1ea446" } new_table_name: "default.Table_05f88ceffe9444ed98eb42878d0238e5" modify_external_catalogs: true
I20251029 08:52:27.151933 1178 tablet_bootstrap.cc:654] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.151901 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 22f9c51cd2ec4689ba2a4d8d466d6716 (DEFAULT_TABLE table=default.Table_66eb18b4419b42f087e8468a9dc15153 [id=306d651c5e4a4909ae5a2c5dcffe3962]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.152338 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f9c51cd2ec4689ba2a4d8d466d6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.155280 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_da1f7a7ff5004099ab9ac18ce51a23d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.155602 757 tablet_service.cc:1505] Processing CreateTablet for tablet 22f9c51cd2ec4689ba2a4d8d466d6716 (DEFAULT_TABLE table=default.Table_66eb18b4419b42f087e8468a9dc15153 [id=306d651c5e4a4909ae5a2c5dcffe3962]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.155864 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da1f7a7ff5004099ab9ac18ce51a23d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.156507 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f9c51cd2ec4689ba2a4d8d466d6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.157478 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_26a3af4de195431a994db99495fdbeb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.158002 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a3af4de195431a994db99495fdbeb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.158918 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5329de213c3e4ae088e1a8d1674186db" } new_table_name: "default.Table_c687b28031f94297b1ef019227b0e921" modify_external_catalogs: true
I20251029 08:52:27.160642 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_6541220eef6e47c1861484bbf1a5f44c" } new_table_name: "default.Table_92f29eaa26a641bfa97d5666fda3609f" modify_external_catalogs: true
I20251029 08:52:27.167598 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2fad7549758649229ad771de926aaf15
I20251029 08:52:27.169368 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_479cc5f9383943c49d34e88baf0b5b28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.170145 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_479cc5f9383943c49d34e88baf0b5b28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.170536 889 tablet_service.cc:1505] Processing CreateTablet for tablet 3498dba075944290bd5ea3c68f6578eb (DEFAULT_TABLE table=default.Table_064cacab68c74e94a8224b2c068aec83 [id=69c514478c9c4cedbe4be3d65e769616]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.172883 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
W20251029 08:52:27.176829 590 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:60007 (request call id 155) took 943 ms (client timeout 1000 ms). Trace:
W20251029 08:52:27.177006 590 rpcz_store.cc:269] 1029 08:52:26.233660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.234650 (+ 990us) service_pool.cc:225] Handling call
1029 08:52:26.239783 (+ 5133us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x616000041254. stack: 00007fb9012e39f6 00007fb9012e37b0 00007fb9002bf3ad 0000000000983f04 0000000000983ecd 00007fb9105ea6bd 00007fb9105ea7b3 00007fb90d1b1441 00007fb914e6b9dd 00007fb914ff2aa8 00007fb90bb05ca6 00007fb90bb059d5 00007fb90828227e 00007fb90828115e 00007fb908287395 00007fb908287cbf
1029 08:52:27.176806 (+937023us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 08:52:27.161987 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 08:52:27.162069 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
1029 08:52:27.162074 (+ 5us) write_op.cc:435] Acquired schema lock
1029 08:52:27.162076 (+ 2us) tablet.cc:660] Decoding operations
1029 08:52:27.162152 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1029 08:52:27.162161 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1029 08:52:27.162164 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1029 08:52:27.162210 (+ 46us) tablet.cc:699] Row locks acquired
1029 08:52:27.162212 (+ 2us) write_op.cc:260] PREPARE: finished
1029 08:52:27.162237 (+ 25us) write_op.cc:270] Start()
1029 08:52:27.162260 (+ 23us) write_op.cc:276] Timestamp: P: 1761727947162194 usec, L: 0
1029 08:52:27.162264 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 08:52:27.162379 (+ 115us) log.cc:844] Serialized 527 byte log entry
1029 08:52:27.167511 (+ 5132us) op_driver.cc:464] REPLICATION: finished
1029 08:52:27.168090 (+ 579us) write_op.cc:301] APPLY: starting
1029 08:52:27.168103 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1029 08:52:27.168140 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
1029 08:52:27.168162 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 08:52:27.168207 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 08:52:27.168236 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 08:52:27.168244 (+ 8us) write_op.cc:312] APPLY: finished
1029 08:52:27.168308 (+ 64us) log.cc:844] Serialized 23 byte log entry
1029 08:52:27.168354 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:27.168366 (+ 12us) write_op.cc:454] Released schema lock
1029 08:52:27.168371 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1029 08:52:27.168458 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1029 08:52:27.168464 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":8778624,"child_traces":[["op",{"apply.queue_time_us":533,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":360,"num_ops":1,"prepare.queue_time_us":69743,"prepare.run_cpu_time_us":449,"prepare.run_wall_time_us":447,"replication_time_us":5233}]]}
I20251029 08:52:27.177829 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3c395564e2fb41c6b92c0abda8d3c16f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.178431 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c395564e2fb41c6b92c0abda8d3c16f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.179711 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20006242722c4134b53340632af61698 consensus state has no leader
W20251029 08:52:27.180833 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d37fdd5241d34200b5d2f7f57a2f9607 consensus state has no leader
I20251029 08:52:27.181344 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98da77093ed14699a141e885b30e3088" } new_table_name: "default.Table_a16e3c2009a242a5af6656d965921df9" modify_external_catalogs: true
I20251029 08:52:27.184199 591 catalog_manager.cc:6719] ReplaceTablet: tablet 2fad7549758649229ad771de926aaf15 deleted and replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68
W20251029 08:52:27.185211 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef67d5cbc902492fb1252229b4b7f28f consensus state has no leader
I20251029 08:52:27.185370 905 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fad7549758649229ad771de926aaf15 with delete_type TABLET_DATA_DELETED (replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.186415 1183 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.189502 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_09a6e417f01d453c95b382fa9533c0fc [id=f1436eb4acb3417e8a547ef8f0203d6f] alter complete (version 1)
I20251029 08:52:27.191258 1183 ts_tablet_manager.cc:1403] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.004s
I20251029 08:52:27.194083 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:27.194360 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f21470e046804cd0b003d632d05de8b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.194967 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f21470e046804cd0b003d632d05de8b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.195621 1178 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.195809 1178 ts_tablet_manager.cc:1403] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251029 08:52:27.196662 1178 raft_consensus.cc:359] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.196899 1178 raft_consensus.cc:385] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.197001 1178 raft_consensus.cc:740] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.197629 2458 raft_consensus.cc:493] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.197571 905 tablet_service.cc:1505] Processing CreateTablet for tablet 67bcb015ada846e3914b9f430c744d33 (DEFAULT_TABLE table=default.Table_7387b39af2fe4efa908305b60633960f [id=f191aa3715214fa3ac98dde3e3ba3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.197731 2458 raft_consensus.cc:515] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.198315 2458 leader_election.cc:290] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.197371 1178 consensus_queue.cc:260] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.199285 1178 ts_tablet_manager.cc:1434] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20251029 08:52:27.200346 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2dbaf0f3d3242b388197299431cf86d consensus state has no leader
I20251029 08:52:27.201328 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2d36524247274c099382f1d22aa6dd76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.202080 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d36524247274c099382f1d22aa6dd76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.202339 1178 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.204368 1178 tablet_bootstrap.cc:654] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.205765 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4dfe9cad106348adb513ca14b0bd54bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.206344 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dfe9cad106348adb513ca14b0bd54bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.203584 1183 raft_consensus.cc:359] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.216641 1183 raft_consensus.cc:385] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.217677 1183 raft_consensus.cc:740] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.220845 2295 consensus_queue.cc:1048] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:27.221480 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:27.219113 1183 consensus_queue.cc:260] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.223049 1183 ts_tablet_manager.cc:1434] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20251029 08:52:27.223376 1183 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.223773 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3bd45ebf7bd64e5cbb19662e966bb925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.224493 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bd45ebf7bd64e5cbb19662e966bb925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.232613 861 consensus_peers.cc:597] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7be4bcf9e3324bd884e946fae20648fd. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.233314 1183 tablet_bootstrap.cc:654] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.237579 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c2072c16fce43278f1e32b9672187e7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.238022 895 tablet_service.cc:1505] Processing CreateTablet for tablet bdfda0866a614165824d1a3f7ad89af7 (DEFAULT_TABLE table=default.Table_39fcfbbc91a84ece9939c873e4a33e9d [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.238194 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fad7549758649229ad771de926aaf15 with delete_type TABLET_DATA_DELETED (replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.238490 749 tablet_service.cc:1505] Processing CreateTablet for tablet 67bcb015ada846e3914b9f430c744d33 (DEFAULT_TABLE table=default.Table_7387b39af2fe4efa908305b60633960f [id=f191aa3715214fa3ac98dde3e3ba3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.238754 747 tablet_service.cc:1552] Processing DeleteTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.238909 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcb015ada846e3914b9f430c744d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.239784 893 tablet_service.cc:1552] Processing DeleteTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.244570 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e321154841c14ccb8ef00074b1573785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.245329 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e321154841c14ccb8ef00074b1573785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.102718 1974 ts_tablet_manager.cc:1929] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:27.247745 1974 log.cc:1199] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e687f593eead488db4bb445a63beeec7
I20251029 08:52:27.248327 1974 ts_tablet_manager.cc:1950] T e687f593eead488db4bb445a63beeec7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:27.249366 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 995) took 3134 ms. Trace:
I20251029 08:52:27.249531 1974 rpcz_store.cc:276] 1029 08:52:24.114921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.114978 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:26.975994 (+2861016us) ts_tablet_manager.cc:1151] Deleting tablet e687f593eead488db4bb445a63beeec7
1029 08:52:26.976005 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:26.988918 (+ 12913us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.102588 (+113670us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.249345 (+146757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2772891}
I20251029 08:52:27.250154 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e687f593eead488db4bb445a63beeec7 (table default.Table_d707b992dd69414897e525d86fb4b002 [id=a23a597c8fd343769b517507845104ff]) successfully deleted
I20251029 08:52:27.123903 896 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee83f5526f74ba2a344edd97db94050 (DEFAULT_TABLE table=default.Table_ac500003d4a84f739ff1c0ac8d40e93c [id=ff2c0066d55c4ce5a52c1def8963d70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.126477 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf9e36c47be4509a998be8e7138475b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.134938 897 tablet_service.cc:1505] Processing CreateTablet for tablet 22f9c51cd2ec4689ba2a4d8d466d6716 (DEFAULT_TABLE table=default.Table_66eb18b4419b42f087e8468a9dc15153 [id=306d651c5e4a4909ae5a2c5dcffe3962]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.252892 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f1dca37ca0fc4bc38418e729d7fbff0b" } modify_external_catalogs: true
I20251029 08:52:27.253115 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f1dca37ca0fc4bc38418e729d7fbff0b" } modify_external_catalogs: true
I20251029 08:52:27.256711 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_70f74ef51bd84663a6a30cbd00c63bb2" } new_table_name: "default.Table_c08dc513f0834dd2b095f48146990078" modify_external_catalogs: true
I20251029 08:52:27.262813 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4cf1783778c64509b2a98c717f0477a2" } modify_external_catalogs: true
I20251029 08:52:27.263095 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4cf1783778c64509b2a98c717f0477a2" } modify_external_catalogs: true
I20251029 08:52:27.263640 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet bd357a723a1d4bc1aa4b480c8a29ef03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.264582 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fe2750c35f884a3eb6ecce08b9a7aa7c" } modify_external_catalogs: true
I20251029 08:52:27.264825 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fe2750c35f884a3eb6ecce08b9a7aa7c" } modify_external_catalogs: true
I20251029 08:52:27.266251 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b60ce497df09480daa025d919c6509e3" } modify_external_catalogs: true
I20251029 08:52:27.266520 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b60ce497df09480daa025d919c6509e3" } modify_external_catalogs: true
I20251029 08:52:27.268033 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fad7549758649229ad771de926aaf15 with delete_type TABLET_DATA_DELETED (replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.268613 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.269129 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.270239 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.270757 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.271251 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.271737 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.272470 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 3498dba075944290bd5ea3c68f6578eb (DEFAULT_TABLE table=default.Table_064cacab68c74e94a8224b2c068aec83 [id=69c514478c9c4cedbe4be3d65e769616]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.272866 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3498dba075944290bd5ea3c68f6578eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.274076 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f744556dccf348fd8b92071f224730aa
I20251029 08:52:27.275301 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cea6d85081f246788ba29bb4a9c9a4e8
I20251029 08:52:27.275975 894 tablet_service.cc:1552] Processing DeleteTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.276261 893 tablet_service.cc:1552] Processing DeleteTablet for tablet f744556dccf348fd8b92071f224730aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.277160 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 38032950744b4426927157fad0962dfb
I20251029 08:52:27.277897 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 38032950744b4426927157fad0962dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:27.278541 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2c7388debdf4a0da6130a385a495245 has no consensus state
I20251029 08:52:27.281347 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.281785 747 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf9e36c47be4509a998be8e7138475b with delete_type TABLET_DATA_DELETED (replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.282167 747 tablet_service.cc:1552] Processing DeleteTablet for tablet eeabf8f23751447aa9af02363c40361d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.282538 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.283113 747 tablet_service.cc:1505] Processing CreateTablet for tablet bdfda0866a614165824d1a3f7ad89af7 (DEFAULT_TABLE table=default.Table_39fcfbbc91a84ece9939c873e4a33e9d [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.283551 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfda0866a614165824d1a3f7ad89af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.283775 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:27.284207 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_8d966fcf45004abdba1b04d1722be5c5 [id=32fdba980db34781a7f359080cb1e56c]). Expected version 2 got 0
I20251029 08:52:27.289002 1178 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.289191 1178 ts_tablet_manager.cc:1403] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20251029 08:52:27.290432 1178 raft_consensus.cc:359] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.290562 1974 tablet_replica.cc:333] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:27.290755 1178 raft_consensus.cc:385] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.290864 1178 raft_consensus.cc:740] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.290899 1974 raft_consensus.cc:2243] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:27.291328 1974 raft_consensus.cc:2272] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.291981 1974 ts_tablet_manager.cc:1916] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.292373 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 46bcc468ce964610a857bc2b11b5b3e6
I20251029 08:52:27.292320 1178 consensus_queue.cc:260] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.293013 1178 ts_tablet_manager.cc:1434] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:27.293316 1178 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.294842 1183 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.295025 1183 ts_tablet_manager.cc:1403] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.003s
I20251029 08:52:27.295238 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.295856 1183 raft_consensus.cc:359] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.296109 1183 raft_consensus.cc:385] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.296206 1183 raft_consensus.cc:740] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.296473 1183 consensus_queue.cc:260] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.297086 1183 ts_tablet_manager.cc:1434] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:27.297405 1183 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.298802 1183 tablet_bootstrap.cc:654] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.299293 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 67bcb015ada846e3914b9f430c744d33 (DEFAULT_TABLE table=default.Table_7387b39af2fe4efa908305b60633960f [id=f191aa3715214fa3ac98dde3e3ba3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.299710 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f744556dccf348fd8b92071f224730aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.300006 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.300032 1044 tablet_service.cc:1505] Processing CreateTablet for tablet bdfda0866a614165824d1a3f7ad89af7 (DEFAULT_TABLE table=default.Table_39fcfbbc91a84ece9939c873e4a33e9d [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.300182 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 38032950744b4426927157fad0962dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.300209 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.304141 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfda0866a614165824d1a3f7ad89af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.300503 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c2072c16fce43278f1e32b9672187e7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.304430 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcb015ada846e3914b9f430c744d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.307904 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_8d966fcf45004abdba1b04d1722be5c5 [id=32fdba980db34781a7f359080cb1e56c]). Expected version 2 got 0
I20251029 08:52:27.313721 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.314150 1178 tablet_bootstrap.cc:654] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.315312 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f744556dccf348fd8b92071f224730aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.315733 745 tablet_service.cc:1552] Processing DeleteTablet for tablet cea6d85081f246788ba29bb4a9c9a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.316107 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 38032950744b4426927157fad0962dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.316658 745 tablet_service.cc:1505] Processing CreateTablet for tablet 977d116f402f425892ea6202d91b7c59 (DEFAULT_TABLE table=default.Table_2b1a82f91b4542a7b759c6efd549e4ba [id=41750b56353e4784ae21b05d170cb6a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.317058 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977d116f402f425892ea6202d91b7c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.320363 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 46bcc468ce964610a857bc2b11b5b3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:27.323128 718 leader_election.cc:343] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:27.323395 718 leader_election.cc:343] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.323604 718 leader_election.cc:304] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.323870 2458 raft_consensus.cc:2749] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.326396 893 tablet_service.cc:1552] Processing DeleteTablet for tablet e687f593eead488db4bb445a63beeec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:24 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.326926 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aa4f97075b3941aaa7cda3f9413e867f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.327664 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0ef904c2fb4dfeadbf56ae659c4e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.328118 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.328924 2475 raft_consensus.cc:493] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.329062 2475 raft_consensus.cc:515] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.329622 2475 leader_election.cc:290] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.329453 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:27.329802 996 consensus_peers.cc:597] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af38662420d14ac1a7cb5fd03fc1b9d8. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:27.333104 720 leader_election.cc:343] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.333209 720 leader_election.cc:304] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.333600 2458 raft_consensus.cc:2749] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.334170 2475 raft_consensus.cc:493] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.334285 2475 raft_consensus.cc:515] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.334717 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a355c68ecfb45ff8c396873d91268a8" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.334818 2475 leader_election.cc:290] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.335702 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "254e62d88c3242a0a3ac7470a8005d74" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.338279 888 tablet_service.cc:1505] Processing CreateTablet for tablet 977d116f402f425892ea6202d91b7c59 (DEFAULT_TABLE table=default.Table_2b1a82f91b4542a7b759c6efd549e4ba [id=41750b56353e4784ae21b05d170cb6a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.346280 998 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7a355c68ecfb45ff8c396873d91268a8
W20251029 08:52:27.346710 998 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 254e62d88c3242a0a3ac7470a8005d74
W20251029 08:52:27.347786 996 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.349509 996 leader_election.cc:304] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.351428 2475 raft_consensus.cc:2749] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:27.352057 996 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.352579 996 leader_election.cc:304] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.353082 2475 raft_consensus.cc:2749] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.356004 1183 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.356498 1183 ts_tablet_manager.cc:1403] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.003s
I20251029 08:52:27.357331 1183 raft_consensus.cc:359] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.357695 2458 raft_consensus.cc:493] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.357568 1183 raft_consensus.cc:385] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.358060 1183 raft_consensus.cc:740] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.358312 1183 consensus_queue.cc:260] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.358903 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_588b8eaf2cad4b739475473bec43d9cf" } modify_external_catalogs: true
I20251029 08:52:27.359133 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_588b8eaf2cad4b739475473bec43d9cf" } modify_external_catalogs: true
I20251029 08:52:27.363098 893 tablet_service.cc:1505] Processing CreateTablet for tablet 834a9dacf9e94263b0406c28adcdb916 (DEFAULT_TABLE table=default.Table_683961d19f454703a28fc07f074d028d [id=80b753b24a6b4927b21f7f83059be071]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.357823 2458 raft_consensus.cc:515] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.373373 2458 leader_election.cc:290] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.375092 1183 ts_tablet_manager.cc:1434] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20251029 08:52:27.375654 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0bd490da0b0e47fc965343c123a27b22" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.375797 745 tablet_service.cc:1505] Processing CreateTablet for tablet 834a9dacf9e94263b0406c28adcdb916 (DEFAULT_TABLE table=default.Table_683961d19f454703a28fc07f074d028d [id=80b753b24a6b4927b21f7f83059be071]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.376248 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834a9dacf9e94263b0406c28adcdb916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.380312 1183 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.380404 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_36f1ed1c59f947ceaba03af7ec3bf4e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.380540 861 consensus_peers.cc:597] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.381381 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb3e353385b14cfaa0d01935be38e1e9" } new_table_name: "default.Table_755066834bcd4fc7bf4f0b206ddafc9c" modify_external_catalogs: true
W20251029 08:52:27.383005 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f1ed1c59f947ceaba03af7ec3bf4e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.383037 1183 tablet_bootstrap.cc:654] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.383438 1178 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.383628 1178 ts_tablet_manager.cc:1403] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.005s
I20251029 08:52:27.383878 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a4f861ca92fe4bb0af9cb605c5465744" } new_table_name: "default.Table_4ccf023f7a3f4275bfb29b5983db651e" modify_external_catalogs: true
I20251029 08:52:27.384567 1178 raft_consensus.cc:359] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.388073 1178 raft_consensus.cc:385] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.388834 1178 raft_consensus.cc:740] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.389264 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 977d116f402f425892ea6202d91b7c59 (DEFAULT_TABLE table=default.Table_2b1a82f91b4542a7b759c6efd549e4ba [id=41750b56353e4784ae21b05d170cb6a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.389649 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977d116f402f425892ea6202d91b7c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.390633 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0988621e83c64bec83ee43c523b09b20" } new_table_name: "default.Table_17251c1e0e6447b0bb5b5cf8c78e42ed" modify_external_catalogs: true
I20251029 08:52:27.396247 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_02232cd3f46941d3865a4159bf92dc0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.396808 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02232cd3f46941d3865a4159bf92dc0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.398020 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_02ce044326e84d9f807967a0a3e56020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.398489 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02ce044326e84d9f807967a0a3e56020 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.399611 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0bd490da0b0e47fc965343c123a27b22" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.400074 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ea2dc34a3418478cbf34a81ec9dd33e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.400632 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea2dc34a3418478cbf34a81ec9dd33e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.400882 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 834a9dacf9e94263b0406c28adcdb916 (DEFAULT_TABLE table=default.Table_683961d19f454703a28fc07f074d028d [id=80b753b24a6b4927b21f7f83059be071]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.401350 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834a9dacf9e94263b0406c28adcdb916. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.402622 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f8348b6234e4dbe8bdb58edad34ea7b consensus state has no leader
I20251029 08:52:27.403012 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bf9b59ca50fd463eb0768ec552e1244a" } new_table_name: "default.Table_b6d28fc5fdc14b10b3436923d1b86bc1" modify_external_catalogs: true
W20251029 08:52:27.404875 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81236599ea1149bca1d1ef1a84e60c1a consensus state has no leader
I20251029 08:52:27.401012 1176 ts_tablet_manager.cc:1929] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:27.405251 1178 consensus_queue.cc:260] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:27.405753 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 101aff0682d747dca3846bf95bce0a5d consensus state has no leader
I20251029 08:52:27.405964 1176 log.cc:1199] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/657cdaad97e24611a997e26a6293ea5e
W20251029 08:52:27.408828 720 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.410827 894 tablet_service.cc:1505] Processing CreateTablet for tablet e499043bc431454b8b00667c008c46fc (DEFAULT_TABLE table=default.Table_6b04f7a83c444c96ab3363ca2d15709d [id=a0c09bcafe1441d69ac8549aa6e279b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.411404 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5d9527d294c432db2ccf60af84731b0 consensus state has no leader
W20251029 08:52:27.413985 718 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.414125 718 leader_election.cc:304] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.414471 2458 raft_consensus.cc:2749] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.415661 757 tablet_service.cc:1505] Processing CreateTablet for tablet e499043bc431454b8b00667c008c46fc (DEFAULT_TABLE table=default.Table_6b04f7a83c444c96ab3363ca2d15709d [id=a0c09bcafe1441d69ac8549aa6e279b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.416049 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e499043bc431454b8b00667c008c46fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.419297 1178 ts_tablet_manager.cc:1434] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251029 08:52:27.423051 758 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.426683 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977d116f402f425892ea6202d91b7c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.426862 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcb015ada846e3914b9f430c744d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.427038 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3498dba075944290bd5ea3c68f6578eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.428495 1177 tablet_bootstrap.cc:654] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.429221 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfda0866a614165824d1a3f7ad89af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.429826 1178 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.430076 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee83f5526f74ba2a344edd97db94050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.432116 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f9c51cd2ec4689ba2a4d8d466d6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.435127 1179 ts_tablet_manager.cc:1929] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:27.435303 1179 log.cc:1199] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7b5051ff776e4975b917db0393a1f945
I20251029 08:52:27.435421 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5a8ff23a00f145d4943422ba0649f47e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.435806 1179 ts_tablet_manager.cc:1950] T 7b5051ff776e4975b917db0393a1f945 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.435971 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646aaf0453134f3c9306a68b45d1fa64. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.435984 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a8ff23a00f145d4943422ba0649f47e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.436367 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 960f17ab6bed4ef5ad382421d84bf8d7
I20251029 08:52:27.437047 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.440275 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:27.440639 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:27.440979 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b17b82921a624ff5a4c04ddaa4462781"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.441428 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 957) took 4054 ms. Trace:
I20251029 08:52:27.441259 593 catalog_manager.cc:5649] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
W20251029 08:52:27.441660 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17b82921a624ff5a4c04ddaa4462781 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.441802 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]). Expected version 1 got 0
I20251029 08:52:27.445748 1179 rpcz_store.cc:276] 1029 08:52:23.386953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.390201 (+ 3248us) service_pool.cc:225] Handling call
1029 08:52:26.978935 (+3588734us) ts_tablet_manager.cc:1151] Deleting tablet 7b5051ff776e4975b917db0393a1f945
1029 08:52:26.979164 (+ 229us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.313562 (+334398us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.435032 (+121470us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.441407 (+ 6375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3588388}
I20251029 08:52:27.448541 1179 tablet_replica.cc:333] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:27.449168 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834a9dacf9e94263b0406c28adcdb916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.450201 1178 tablet_bootstrap.cc:654] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.452294 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e499043bc431454b8b00667c008c46fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.456769 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 24834985a71349b9865840e9b5537d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.458523 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 41622daeb1e0419fa88cc26c0405c2db from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:27.458854 1176 ts_tablet_manager.cc:1950] T 657cdaad97e24611a997e26a6293ea5e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:27.470595 889 tablet_service.cc:1505] Processing CreateTablet for tablet 804e677973e44db895db9facf7917ba2 (DEFAULT_TABLE table=default.Table_b9c2103e507b4beb9f03f4cbca60976d [id=0fc676aa91e44c65849394caf076b5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.471055 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804e677973e44db895db9facf7917ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.472298 1179 raft_consensus.cc:2243] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:27.472813 1179 raft_consensus.cc:2272] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.473486 1179 ts_tablet_manager.cc:1916] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.473938 1974 ts_tablet_manager.cc:1929] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:27.474047 1974 log.cc:1199] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/10baa0547180493f872a23614cf4c608
I20251029 08:52:27.474426 1974 ts_tablet_manager.cc:1950] T 10baa0547180493f872a23614cf4c608 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:27.475812 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43572296e299444e8456983d5b1982f3" } modify_external_catalogs: true
I20251029 08:52:27.476058 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43572296e299444e8456983d5b1982f3" } modify_external_catalogs: true
I20251029 08:52:27.478780 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7f6426ed22804c409dd431e21ae1d0ca" } modify_external_catalogs: true
I20251029 08:52:27.478993 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7f6426ed22804c409dd431e21ae1d0ca" } modify_external_catalogs: true
I20251029 08:52:27.481987 2475 raft_consensus.cc:493] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.482148 2475 raft_consensus.cc:515] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.482744 2475 leader_election.cc:290] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.481451 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_45635496357a40e4ba3c778e9ca8b7b0" } new_table_name: "default.Table_eb9a1227328d4c7a8ac782e532122e90" modify_external_catalogs: true
I20251029 08:52:27.481730 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7b5051ff776e4975b917db0393a1f945 (table default.Table_671a7306577e49c98f6ec7c4317f2f31 [id=37c83d47af7a4c19a906af5f1ac9e405]) successfully deleted
I20251029 08:52:27.482235 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4db3fb50b14d461aad1b30f1f49045d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.483541 1050 tablet_service.cc:1505] Processing CreateTablet for tablet e499043bc431454b8b00667c008c46fc (DEFAULT_TABLE table=default.Table_6b04f7a83c444c96ab3363ca2d15709d [id=a0c09bcafe1441d69ac8549aa6e279b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.483903 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e499043bc431454b8b00667c008c46fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.489332 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4db3fb50b14d461aad1b30f1f49045d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.477917 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f096ddf6ddeb4b64b780d57471d56d8d" } modify_external_catalogs: true
I20251029 08:52:27.490890 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f096ddf6ddeb4b64b780d57471d56d8d" } modify_external_catalogs: true
I20251029 08:52:27.491005 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.491353 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.491428 1177 tablet_bootstrap.cc:492] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:27.491557 1177 ts_tablet_manager.cc:1403] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.346s user 0.008s sys 0.010s
I20251029 08:52:27.491745 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.492089 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.492260 2475 raft_consensus.cc:493] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.492303 1177 raft_consensus.cc:359] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.492370 2475 raft_consensus.cc:515] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.492472 1177 raft_consensus.cc:385] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.492528 1177 raft_consensus.cc:740] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.492753 1177 consensus_queue.cc:260] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.492939 2475 leader_election.cc:290] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.493332 1177 ts_tablet_manager.cc:1434] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:27.493386 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.493726 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.493798 905 tablet_service.cc:1505] Processing CreateTablet for tablet 592c3013d8104a56a15b4c1f29cdd6d1 (DEFAULT_TABLE table=default.Table_9310e1a601d94499a44877ea5a9dcc27 [id=ca104710a5034c469a32638e95ab88f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.494057 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.494505 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 804e677973e44db895db9facf7917ba2 (DEFAULT_TABLE table=default.Table_b9c2103e507b4beb9f03f4cbca60976d [id=0fc676aa91e44c65849394caf076b5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.494840 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804e677973e44db895db9facf7917ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.496487 2458 raft_consensus.cc:493] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.496624 2458 raft_consensus.cc:515] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.496690 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:27.496869 773 raft_consensus.cc:2468] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:27.497189 2458 leader_election.cc:290] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.497527 747 tablet_service.cc:1505] Processing CreateTablet for tablet 592c3013d8104a56a15b4c1f29cdd6d1 (DEFAULT_TABLE table=default.Table_9310e1a601d94499a44877ea5a9dcc27 [id=ca104710a5034c469a32638e95ab88f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.500203 998 leader_election.cc:304] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:27.500495 998 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2fb380018f46465e82ac6223ecf8dd0d
I20251029 08:52:27.500712 2475 raft_consensus.cc:2804] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:27.500720 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 960f17ab6bed4ef5ad382421d84bf8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.500787 2475 raft_consensus.cc:493] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:27.500849 2475 raft_consensus.cc:3060] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:27.501775 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1033) took 1369 ms. Trace:
I20251029 08:52:27.501855 1176 rpcz_store.cc:276] 1029 08:52:26.132008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.132076 (+ 68us) service_pool.cc:225] Handling call
1029 08:52:26.971858 (+839782us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:26.971864 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.246839 (+274975us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.396382 (+149543us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.501751 (+105369us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3935,"tablet-delete.queue_time_us":839636}
I20251029 08:52:27.502018 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592c3013d8104a56a15b4c1f29cdd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.502183 1176 tablet_replica.cc:333] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:27.502351 1176 raft_consensus.cc:2243] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.502480 1176 raft_consensus.cc:2272] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.503060 1176 ts_tablet_manager.cc:1916] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.503607 2475 raft_consensus.cc:515] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.504374 2475 leader_election.cc:290] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.508143 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:27.508330 783 raft_consensus.cc:3060] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:27.508693 758 tablet_service.cc:1505] Processing CreateTablet for tablet ee2916d23a0a4e42a9a143a47b047d56 (DEFAULT_TABLE table=default.Table_29c6685703864fd8bcb50adca11929d1 [id=27451a477d784e5996dd0c52eec8328a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.508752 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 997) took 3183 ms. Trace:
I20251029 08:52:27.508832 1974 rpcz_store.cc:276] 1029 08:52:24.325526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.325592 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:27.290505 (+2964913us) ts_tablet_manager.cc:1151] Deleting tablet 10baa0547180493f872a23614cf4c608
1029 08:52:27.290519 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.470742 (+180223us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.473861 (+ 3119us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.508729 (+ 34868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2964699}
I20251029 08:52:27.509016 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2916d23a0a4e42a9a143a47b047d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.510879 783 raft_consensus.cc:2468] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:27.513566 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eeabf8f23751447aa9af02363c40361d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.513815 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 592c3013d8104a56a15b4c1f29cdd6d1 (DEFAULT_TABLE table=default.Table_9310e1a601d94499a44877ea5a9dcc27 [id=ca104710a5034c469a32638e95ab88f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.514169 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592c3013d8104a56a15b4c1f29cdd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.515254 745 tablet_service.cc:1505] Processing CreateTablet for tablet 804e677973e44db895db9facf7917ba2 (DEFAULT_TABLE table=default.Table_b9c2103e507b4beb9f03f4cbca60976d [id=0fc676aa91e44c65849394caf076b5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.515609 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804e677973e44db895db9facf7917ba2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.516434 720 leader_election.cc:343] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.520931 998 leader_election.cc:304] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:27.521173 2475 raft_consensus.cc:2804] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:27.521651 2475 raft_consensus.cc:697] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:27.521971 2475 consensus_queue.cc:237] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.524675 2458 raft_consensus.cc:493] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.524802 2458 raft_consensus.cc:515] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.525352 2458 leader_election.cc:290] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:27.530628 1176 ts_tablet_manager.cc:1929] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:27.531009 1176 log.cc:1199] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0ab56045bd814d59831a98618c759af4
I20251029 08:52:27.531957 1176 ts_tablet_manager.cc:1950] T 0ab56045bd814d59831a98618c759af4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:27.534586 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1034) took 1402 ms. Trace:
I20251029 08:52:27.534850 1176 rpcz_store.cc:276] 1029 08:52:26.132293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.132385 (+ 92us) service_pool.cc:225] Handling call
1029 08:52:27.502143 (+1369758us) ts_tablet_manager.cc:1151] Deleting tablet 0ab56045bd814d59831a98618c759af4
1029 08:52:27.502152 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.506107 (+ 3955us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.530255 (+ 24148us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.534479 (+ 4224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369638}
I20251029 08:52:27.536242 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1043) took 1347 ms. Trace:
I20251029 08:52:27.536506 1176 rpcz_store.cc:276] 1029 08:52:26.188952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.189135 (+ 183us) service_pool.cc:225] Handling call
1029 08:52:27.535662 (+1346527us) ts_tablet_manager.cc:1151] Deleting tablet 919e5494b84343639a2b9b81b05f1c2c
1029 08:52:27.535761 (+ 99us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.536141 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346313}
I20251029 08:52:27.537632 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1045) took 1335 ms. Trace:
I20251029 08:52:27.537890 1176 rpcz_store.cc:276] 1029 08:52:26.201855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.201950 (+ 95us) service_pool.cc:225] Handling call
1029 08:52:27.537170 (+1335220us) ts_tablet_manager.cc:1151] Deleting tablet 3c9cdcec8e494050a475defd0cfd119f
1029 08:52:27.537181 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.537615 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334977}
I20251029 08:52:27.538831 1176 tablet_replica.cc:333] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:27.539625 1176 raft_consensus.cc:2243] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.540179 1176 raft_consensus.cc:2272] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.541891 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59ad04f57c29407b9d2420520c095a28" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:27.542343 720 leader_election.cc:343] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.542788 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eeabf8f23751447aa9af02363c40361d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.547232 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.547449 919 raft_consensus.cc:2468] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:27.547659 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb380018f46465e82ac6223ecf8dd0d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.547897 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c74b052ac94eae9e226285504c6b40" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:27.548053 922 raft_consensus.cc:3060] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:27.548134 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59ad04f57c29407b9d2420520c095a28" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.550850 922 raft_consensus.cc:2468] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:27.551934 1177 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:27.553387 1177 tablet_bootstrap.cc:654] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.555089 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592c3013d8104a56a15b4c1f29cdd6d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.555409 718 leader_election.cc:343] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.555534 718 leader_election.cc:304] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.555917 2458 raft_consensus.cc:2749] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:27.556088 718 leader_election.cc:343] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.556174 718 leader_election.cc:304] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.556336 2458 raft_consensus.cc:2749] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.557128 1183 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.557255 1183 ts_tablet_manager.cc:1403] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.177s user 0.000s sys 0.007s
I20251029 08:52:27.557969 1183 raft_consensus.cc:359] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.558151 1183 raft_consensus.cc:385] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.558202 1183 raft_consensus.cc:740] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.558373 1183 consensus_queue.cc:260] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.558848 1183 ts_tablet_manager.cc:1434] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:27.559096 1183 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.561363 2458 raft_consensus.cc:493] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.561497 2458 raft_consensus.cc:515] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.562073 2458 leader_election.cc:290] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.562762 1974 tablet_replica.cc:333] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:27.562956 1974 raft_consensus.cc:2243] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.563118 1974 raft_consensus.cc:2272] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.563601 1974 ts_tablet_manager.cc:1916] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.566093 1177 tablet_bootstrap.cc:492] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:27.566205 1177 ts_tablet_manager.cc:1403] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20251029 08:52:27.566218 2708 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:27.566939 1177 raft_consensus.cc:359] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.567095 1177 raft_consensus.cc:385] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.567149 1177 raft_consensus.cc:740] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.567312 1177 consensus_queue.cc:260] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.569957 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b52314139945dd9c2e6ec899bf06dc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:27.571842 718 leader_election.cc:343] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.574141 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]) successfully deleted
I20251029 08:52:27.578609 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_be08b1df1890403fbf34357b83147e36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.579509 1177 ts_tablet_manager.cc:1434] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251029 08:52:27.579788 1177 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:27.580621 1178 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.580768 1178 ts_tablet_manager.cc:1403] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.006s
I20251029 08:52:27.581583 1177 tablet_bootstrap.cc:654] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.581467 1178 raft_consensus.cc:359] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.581638 1178 raft_consensus.cc:385] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.581700 1178 raft_consensus.cc:740] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.581925 1178 consensus_queue.cc:260] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.584518 889 tablet_service.cc:1505] Processing CreateTablet for tablet 208530f494ed41c19ab024aeab41445a (DEFAULT_TABLE table=default.Table_67c25e495c4d4292a9b38ecba68278e3 [id=c3ebbef704404eaeac9b87de9c8958a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.584622 899 tablet_service.cc:1505] Processing CreateTablet for tablet ee2916d23a0a4e42a9a143a47b047d56 (DEFAULT_TABLE table=default.Table_29c6685703864fd8bcb50adca11929d1 [id=27451a477d784e5996dd0c52eec8328a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.584867 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208530f494ed41c19ab024aeab41445a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.585024 888 tablet_service.cc:1505] Processing CreateTablet for tablet 486a3fc1b6b24028b23fc8ab92832358 (DEFAULT_TABLE table=default.Table_66e4206742ae4022be3b1011f9bb49be [id=b6b631b6f4664a17889946119510dcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.585309 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486a3fc1b6b24028b23fc8ab92832358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.588233 886 tablet_service.cc:1505] Processing CreateTablet for tablet ec4a6972e983445bae78b31a7f1ad718 (DEFAULT_TABLE table=default.Table_2471fce4e7254548909d80a8b0c78ea9 [id=d4db2c97f8024ef6a2ac84cb470116ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.588542 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4a6972e983445bae78b31a7f1ad718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.592272 1179 ts_tablet_manager.cc:1929] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:27.592265 897 tablet_service.cc:1505] Processing CreateTablet for tablet 46583c1d493a48e398142928a5ed26ab (DEFAULT_TABLE table=default.Table_45635496357a40e4ba3c778e9ca8b7b0 [id=862c691051a249d998152925cb0a7467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.592398 1179 log.cc:1199] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/af38662420d14ac1a7cb5fd03fc1b9d8
I20251029 08:52:27.592561 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46583c1d493a48e398142928a5ed26ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.594028 896 tablet_service.cc:1505] Processing CreateTablet for tablet 56f9308235f145ee838a55434f772bfa (DEFAULT_TABLE table=default.Table_e15c46fe42ab4546913d8f009b074838 [id=5676f1f1a4c84e52a984cb3a60a8c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.594305 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f9308235f145ee838a55434f772bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.594296 1179 ts_tablet_manager.cc:1950] T af38662420d14ac1a7cb5fd03fc1b9d8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.594586 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 958) took 4207 ms. Trace:
I20251029 08:52:27.594687 1179 rpcz_store.cc:276] 1029 08:52:23.387075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:23.402345 (+ 15270us) service_pool.cc:225] Handling call
1029 08:52:27.448494 (+4046149us) ts_tablet_manager.cc:1151] Deleting tablet af38662420d14ac1a7cb5fd03fc1b9d8
1029 08:52:27.448503 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.477012 (+ 28509us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.592161 (+115149us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.594570 (+ 2409us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2304,"tablet-delete.queue_time_us":4045786}
I20251029 08:52:27.595180 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 959) took 3569 ms. Trace:
I20251029 08:52:27.595256 1179 rpcz_store.cc:276] 1029 08:52:24.025869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.025932 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:27.595041 (+3569109us) ts_tablet_manager.cc:1151] Deleting tablet 9493653c2e354458923e9e8dd0da0c60
1029 08:52:27.595054 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.595163 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568902}
I20251029 08:52:27.595594 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 960) took 3569 ms. Trace:
I20251029 08:52:27.595676 1179 rpcz_store.cc:276] 1029 08:52:24.025973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.026190 (+ 217us) service_pool.cc:225] Handling call
1029 08:52:27.595489 (+3569299us) ts_tablet_manager.cc:1151] Deleting tablet d5bc2f2f5d6e418999fa40a6fadbbc71
1029 08:52:27.595499 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.595579 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3569165}
I20251029 08:52:27.596004 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 961) took 3569 ms. Trace:
I20251029 08:52:27.596096 1179 rpcz_store.cc:276] 1029 08:52:24.026058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.026346 (+ 288us) service_pool.cc:225] Handling call
1029 08:52:27.595903 (+3569557us) ts_tablet_manager.cc:1151] Deleting tablet a92cc4a93a584dffaafe21316fe8dc06
1029 08:52:27.595913 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.595988 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3569391}
I20251029 08:52:27.596418 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 962) took 3570 ms. Trace:
I20251029 08:52:27.596501 1179 rpcz_store.cc:276] 1029 08:52:24.026136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.026545 (+ 409us) service_pool.cc:225] Handling call
1029 08:52:27.596317 (+3569772us) ts_tablet_manager.cc:1151] Deleting tablet fb22af7515084e099b41f9c1c92c067a
1029 08:52:27.596327 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.596405 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3569622}
I20251029 08:52:27.597801 1178 ts_tablet_manager.cc:1434] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251029 08:52:27.598382 2712 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:27.600447 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2916d23a0a4e42a9a143a47b047d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.604277 1178 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.605592 1178 tablet_bootstrap.cc:654] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.596798 1179 tablet_replica.cc:333] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:27.610077 1179 raft_consensus.cc:2243] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.610309 1179 raft_consensus.cc:2272] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.610874 1179 ts_tablet_manager.cc:1916] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.613360 1183 tablet_bootstrap.cc:654] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:27.617648 861 consensus_peers.cc:597] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa4f97075b3941aaa7cda3f9413e867f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.619614 2359 raft_consensus.cc:493] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.619764 2359 raft_consensus.cc:515] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.620361 2359 leader_election.cc:290] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:27.621320 998 leader_election.cc:343] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e2b87d133f8a4bd1877e07a1a8758147
I20251029 08:52:27.623045 1178 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.623229 1178 ts_tablet_manager.cc:1403] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20251029 08:52:27.624199 1178 raft_consensus.cc:359] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.624372 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 21d409c2e24947beb99351da46a5aef9
I20251029 08:52:27.625588 1178 raft_consensus.cc:385] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.625746 1178 raft_consensus.cc:740] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.626006 1178 consensus_queue.cc:260] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.626765 888 tablet_service.cc:1505] Processing CreateTablet for tablet af3f21823277420f8f9897d3956c6a4f (DEFAULT_TABLE table=default.Table_4a5189be4b234b599b25a1f049a1b849 [id=0b388756e3c64793b18b11e52523f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.627185 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3f21823277420f8f9897d3956c6a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.628221 1178 ts_tablet_manager.cc:1434] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:27.628633 1178 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.630487 905 tablet_service.cc:1505] Processing CreateTablet for tablet 252faeca036f416493fd050d76a678f7 (DEFAULT_TABLE table=default.Table_d03243f8356b4f47b90023ee624bbb95 [id=744bd6bb788040cfa21a5905d630753d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.630817 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 21d409c2e24947beb99351da46a5aef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.635128 1183 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.635281 1183 ts_tablet_manager.cc:1403] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.003s
I20251029 08:52:27.636057 1183 raft_consensus.cc:359] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.636289 1183 raft_consensus.cc:385] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.636386 1183 raft_consensus.cc:740] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.636658 1183 consensus_queue.cc:260] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.637274 1183 ts_tablet_manager.cc:1434] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:27.637598 1183 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.637657 1178 tablet_bootstrap.cc:654] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.638393 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:27.625468 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6c82dc7545d54dc7995dc1a15f03d6b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.640615 1974 ts_tablet_manager.cc:1929] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:27.640925 1974 log.cc:1199] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a73a891fb6034f989d29a3f837303d84
I20251029 08:52:27.642710 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:27.643973 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be08b1df1890403fbf34357b83147e36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.642010 2458 raft_consensus.cc:493] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.644279 2458 raft_consensus.cc:515] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.645004 2458 leader_election.cc:290] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.645761 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b20d53d6214bdc81ca205dbf922d7b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:27.646265 718 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.647305 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 0b61c710897f43f798d88d92c4676d0d from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:27.647776 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_096667619d1847a998aa237bb19f9024" } modify_external_catalogs: true
I20251029 08:52:27.649613 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 657cdaad97e24611a997e26a6293ea5e (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281]) successfully deleted
W20251029 08:52:27.649775 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 143593us
I20251029 08:52:27.649926 2359 raft_consensus.cc:493] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.650059 2359 raft_consensus.cc:515] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.650645 2359 leader_election.cc:290] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.650821 758 tablet_service.cc:1505] Processing CreateTablet for tablet 56f9308235f145ee838a55434f772bfa (DEFAULT_TABLE table=default.Table_e15c46fe42ab4546913d8f009b074838 [id=5676f1f1a4c84e52a984cb3a60a8c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.651193 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f9308235f145ee838a55434f772bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.652602 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet af38662420d14ac1a7cb5fd03fc1b9d8 (table default.Table_69a638baaadc493f9439102862bb7607 [id=14f726f4d8104eaf99f5dbb58f9ce22c]) successfully deleted
I20251029 08:52:27.652937 747 tablet_service.cc:1505] Processing CreateTablet for tablet ec4a6972e983445bae78b31a7f1ad718 (DEFAULT_TABLE table=default.Table_2471fce4e7254548909d80a8b0c78ea9 [id=d4db2c97f8024ef6a2ac84cb470116ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.653054 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0ab56045bd814d59831a98618c759af4 (table default.Table_29abde71edfc496a8c61702f28f25c63 [id=771dd3dd3a224a8599ab86bc7801dfa3]) successfully deleted
I20251029 08:52:27.653353 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4a6972e983445bae78b31a7f1ad718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.656137 757 tablet_service.cc:1505] Processing CreateTablet for tablet 208530f494ed41c19ab024aeab41445a (DEFAULT_TABLE table=default.Table_67c25e495c4d4292a9b38ecba68278e3 [id=c3ebbef704404eaeac9b87de9c8958a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.656575 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208530f494ed41c19ab024aeab41445a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.656567 589 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c3da4b29e3c4208ae2109789f994c1e from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:27.656960 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 919e5494b84343639a2b9b81b05f1c2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 919e5494b84343639a2b9b81b05f1c2c already in progress: creating tablet
W20251029 08:52:27.657200 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3c9cdcec8e494050a475defd0cfd119f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c9cdcec8e494050a475defd0cfd119f already in progress: creating tablet
I20251029 08:52:27.657606 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0cc95257c937454aa6cd55b5a9394a56
I20251029 08:52:27.660595 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_096667619d1847a998aa237bb19f9024" } modify_external_catalogs: true
I20251029 08:52:27.660753 750 tablet_service.cc:1505] Processing CreateTablet for tablet 252faeca036f416493fd050d76a678f7 (DEFAULT_TABLE table=default.Table_d03243f8356b4f47b90023ee624bbb95 [id=744bd6bb788040cfa21a5905d630753d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.660955 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.661053 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252faeca036f416493fd050d76a678f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.661239 749 tablet_service.cc:1505] Processing CreateTablet for tablet 46583c1d493a48e398142928a5ed26ab (DEFAULT_TABLE table=default.Table_45635496357a40e4ba3c778e9ca8b7b0 [id=862c691051a249d998152925cb0a7467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.661535 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46583c1d493a48e398142928a5ed26ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.662034 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46583c1d493a48e398142928a5ed26ab consensus state has no leader
I20251029 08:52:27.662406 1974 ts_tablet_manager.cc:1950] T a73a891fb6034f989d29a3f837303d84 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:27.663398 886 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a73e1aad5479eb07282c642297dc0 (DEFAULT_TABLE table=default.Table_ea52c1c186c8469b9a0a7bfba7654b5d [id=157bea50919a40eb9532dbd490291bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.664119 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 21d409c2e24947beb99351da46a5aef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.664219 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5d8159a04f9f454c8d3d5b4472097608" } new_table_name: "default.Table_f05841b1b5d941c28eb6d6c0a79e8e69" modify_external_catalogs: true
I20251029 08:52:27.664458 752 tablet_service.cc:1505] Processing CreateTablet for tablet af3f21823277420f8f9897d3956c6a4f (DEFAULT_TABLE table=default.Table_4a5189be4b234b599b25a1f049a1b849 [id=0b388756e3c64793b18b11e52523f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.664737 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3f21823277420f8f9897d3956c6a4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.665736 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c82dc7545d54dc7995dc1a15f03d6b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.666194 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_39fcfbbc91a84ece9939c873e4a33e9d" } new_table_name: "default.Table_6fe654db95e54d37a885e5e31bc35b37" modify_external_catalogs: true
I20251029 08:52:27.668238 764 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a73e1aad5479eb07282c642297dc0 (DEFAULT_TABLE table=default.Table_ea52c1c186c8469b9a0a7bfba7654b5d [id=157bea50919a40eb9532dbd490291bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.668556 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a73e1aad5479eb07282c642297dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.669090 745 tablet_service.cc:1505] Processing CreateTablet for tablet 486a3fc1b6b24028b23fc8ab92832358 (DEFAULT_TABLE table=default.Table_66e4206742ae4022be3b1011f9bb49be [id=b6b631b6f4664a17889946119510dcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.669353 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486a3fc1b6b24028b23fc8ab92832358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.673659 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_153c84094e934e56abac30a2f8d6d02b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.679560 998 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0ab56045bd814d59831a98618c759af4
I20251029 08:52:27.679649 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 1fb9c8c7286149128c2741635c242341
I20251029 08:52:27.680595 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb9c8c7286149128c2741635c242341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.681728 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d38046985bde419896eda670b4597795" } new_table_name: "default.Table_4e0e56581a594a94a0d9ca96193d96f8" modify_external_catalogs: true
I20251029 08:52:27.682816 1179 ts_tablet_manager.cc:1929] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:27.682919 1179 log.cc:1199] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e687f593eead488db4bb445a63beeec7
I20251029 08:52:27.683238 1179 ts_tablet_manager.cc:1950] T e687f593eead488db4bb445a63beeec7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.684233 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 963) took 3557 ms. Trace:
I20251029 08:52:27.685396 1179 rpcz_store.cc:276] 1029 08:52:24.127267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.127357 (+ 90us) service_pool.cc:225] Handling call
1029 08:52:27.596750 (+3469393us) ts_tablet_manager.cc:1151] Deleting tablet e687f593eead488db4bb445a63beeec7
1029 08:52:27.596760 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.615853 (+ 19093us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.682714 (+ 66861us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.684215 (+ 1501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3469156}
W20251029 08:52:27.685703 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8623b1f73194d3084b4ac04568858e1 consensus state has no leader
I20251029 08:52:27.688735 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b87d133f8a4bd1877e07a1a8758147" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.686102 1179 tablet_replica.cc:333] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:27.689575 996 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.689693 996 leader_election.cc:304] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.689903 2485 raft_consensus.cc:2749] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.689947 1179 raft_consensus.cc:2243] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.690007 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.690109 1179 raft_consensus.cc:2272] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.690604 1179 ts_tablet_manager.cc:1916] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.691201 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb9c8c7286149128c2741635c242341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.691251 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.691130 745 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.691449 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fdc381a0d949b0b93cef399daa3021 with delete_type TABLET_DATA_DELETED (replaced by tablet 8017e8fe5f634ef0bf7f8be7c8aea033) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.693133 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252faeca036f416493fd050d76a678f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.693451 1183 tablet_bootstrap.cc:654] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.695968 1177 tablet_bootstrap.cc:492] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:27.696167 1177 ts_tablet_manager.cc:1403] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.014s
I20251029 08:52:27.696944 1177 raft_consensus.cc:359] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.697162 1177 raft_consensus.cc:385] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.697248 1177 raft_consensus.cc:740] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.697504 1177 consensus_queue.cc:260] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.698132 1177 ts_tablet_manager.cc:1434] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:27.698408 1177 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:27.699370 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a73e1aad5479eb07282c642297dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.700011 1177 tablet_bootstrap.cc:654] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.705533 1176 ts_tablet_manager.cc:1916] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.712778 1178 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:27.713022 1178 ts_tablet_manager.cc:1403] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20251029 08:52:27.713992 1179 ts_tablet_manager.cc:1929] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:27.713912 1178 raft_consensus.cc:359] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.714080 1178 raft_consensus.cc:385] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.714133 1178 raft_consensus.cc:740] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.714504 1178 consensus_queue.cc:260] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.714857 1179 log.cc:1199] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/10baa0547180493f872a23614cf4c608
I20251029 08:52:27.715210 1179 ts_tablet_manager.cc:1950] T 10baa0547180493f872a23614cf4c608 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.715935 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 964) took 3388 ms. Trace:
I20251029 08:52:27.716069 1179 rpcz_store.cc:276] 1029 08:52:24.327298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.327349 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:27.686058 (+3358709us) ts_tablet_manager.cc:1151] Deleting tablet 10baa0547180493f872a23614cf4c608
1029 08:52:27.686068 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.703032 (+ 16964us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.713886 (+ 10854us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.715916 (+ 2030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3358543}
I20251029 08:52:27.717537 1177 tablet_bootstrap.cc:492] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:27.717717 1177 ts_tablet_manager.cc:1403] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251029 08:52:27.718425 1177 raft_consensus.cc:359] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.718632 1177 raft_consensus.cc:385] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.718730 1177 raft_consensus.cc:740] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.718995 1177 consensus_queue.cc:260] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.719600 1177 ts_tablet_manager.cc:1434] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:27.720196 1179 tablet_replica.cc:333] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:27.720237 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bdfda0866a614165824d1a3f7ad89af7 consensus state has no leader
I20251029 08:52:27.720481 1177 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:27.720659 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_691d182612d74f709b0e5d106e4a5e11" } modify_external_catalogs: true
I20251029 08:52:27.720726 1179 raft_consensus.cc:2243] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.720887 1179 raft_consensus.cc:2272] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.663110 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1000) took 3074 ms. Trace:
I20251029 08:52:27.721375 1974 rpcz_store.cc:276] 1029 08:52:24.588393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.588453 (+ 60us) service_pool.cc:225] Handling call
1029 08:52:27.509185 (+2920732us) ts_tablet_manager.cc:1151] Deleting tablet a73a891fb6034f989d29a3f837303d84
1029 08:52:27.562704 (+ 53519us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.571037 (+ 8333us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.640301 (+ 69264us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.663058 (+ 22757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920502}
I20251029 08:52:27.722589 1178 ts_tablet_manager.cc:1434] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 08:52:27.722946 1178 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:27.723703 1179 ts_tablet_manager.cc:1916] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.724453 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1002) took 2708 ms. Trace:
I20251029 08:52:27.724597 1974 rpcz_store.cc:276] 1029 08:52:25.016176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.016323 (+ 147us) service_pool.cc:225] Handling call
1029 08:52:27.724298 (+2707975us) ts_tablet_manager.cc:1151] Deleting tablet 25bd82e3549442ada2503518037127fc
1029 08:52:27.724308 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.724425 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707770}
I20251029 08:52:27.725764 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.731959 2458 raft_consensus.cc:493] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.732123 2458 raft_consensus.cc:515] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.732681 2458 leader_election.cc:290] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:27.733107 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1003) took 2706 ms. Trace:
I20251029 08:52:27.733260 1974 rpcz_store.cc:276] 1029 08:52:25.026467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.026529 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:27.732865 (+2706336us) ts_tablet_manager.cc:1151] Deleting tablet 5f67e7b28b4240618e1817bb47d5633d
1029 08:52:27.732905 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.733064 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705159}
I20251029 08:52:27.733707 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9907d044daa145168417371c2bd66544" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.733965 1974 tablet_replica.cc:333] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:27.733935 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_153c84094e934e56abac30a2f8d6d02b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.734181 718 leader_election.cc:343] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.734397 1974 raft_consensus.cc:2243] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.734851 1974 raft_consensus.cc:2272] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.735294 2485 consensus_queue.cc:1048] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:27.736811 1183 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.737002 1183 ts_tablet_manager.cc:1403] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20251029 08:52:27.737872 1183 raft_consensus.cc:359] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.738099 1183 raft_consensus.cc:385] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.738200 1183 raft_consensus.cc:740] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.738448 1183 consensus_queue.cc:260] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.739050 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c687b28031f94297b1ef019227b0e921" } new_table_name: "default.Table_a597495f654c4400b4229478be067582" modify_external_catalogs: true
I20251029 08:52:27.742280 1183 ts_tablet_manager.cc:1434] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 08:52:27.742604 1183 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:27.744382 1183 tablet_bootstrap.cc:654] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:27.749118 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 58241.9us
W20251029 08:52:27.749733 996 leader_election.cc:343] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.750180 996 leader_election.cc:304] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.750960 2359 raft_consensus.cc:2749] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.754438 2451 consensus_queue.cc:1048] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:27.763962 861 consensus_peers.cc:597] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba201a54466240129d82a096f9316242. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:27.764600 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9493653c2e354458923e9e8dd0da0c60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9493653c2e354458923e9e8dd0da0c60 already in progress: creating tablet
W20251029 08:52:27.764791 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 74171.6us
W20251029 08:52:27.765229 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.766031 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d5bc2f2f5d6e418999fa40a6fadbbc71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bc2f2f5d6e418999fa40a6fadbbc71 already in progress: creating tablet
W20251029 08:52:27.766299 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a92cc4a93a584dffaafe21316fe8dc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92cc4a93a584dffaafe21316fe8dc06 already in progress: creating tablet
I20251029 08:52:27.766537 1974 ts_tablet_manager.cc:1916] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:27.767781 2451 consensus_queue.cc:1048] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:27.783272 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet fb22af7515084e099b41f9c1c92c067a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb22af7515084e099b41f9c1c92c067a already in progress: creating tablet
I20251029 08:52:27.783818 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b52314139945dd9c2e6ec899bf06dc" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.785192 747 tablet_service.cc:1505] Processing CreateTablet for tablet 473c35faedf04353b1bf8971b268d5f3 (DEFAULT_TABLE table=default.Table_2696186101b6433885b1929577db93d7 [id=96a093af6de342efbb04624f20ff3890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.785606 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473c35faedf04353b1bf8971b268d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.788273 757 tablet_service.cc:1505] Processing CreateTablet for tablet f2afefb80dc14001811ddb811f21fd6d (DEFAULT_TABLE table=default.Table_c5f70a8f965d4999a64b666bffc151c1 [id=7c8eae3dfd3e42a8888990aeadc118a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.788638 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2afefb80dc14001811ddb811f21fd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.792274 764 tablet_service.cc:1505] Processing CreateTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 (DEFAULT_TABLE table=default.Table_ccddc094f01348a5b25ebd260db6d8cd [id=7292f42c9fd548ecb91633f63eed6e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.792382 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_4680189b619a426596baa3e27ee29f57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.792583 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea40ed38c0ba4d2f9045188af226ced5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.792896 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4680189b619a426596baa3e27ee29f57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.793325 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d2361c16d6164246a7b81385f8fcf1c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.793813 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2361c16d6164246a7b81385f8fcf1c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.794849 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
W20251029 08:52:27.794987 582 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 72977.8us
I20251029 08:52:27.795179 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b20d53d6214bdc81ca205dbf922d7b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:27.795522 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 25bd82e3549442ada2503518037127fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bd82e3549442ada2503518037127fc already in progress: creating tablet
I20251029 08:52:27.795634 1050 tablet_service.cc:1505] Processing CreateTablet for tablet ee2916d23a0a4e42a9a143a47b047d56 (DEFAULT_TABLE table=default.Table_29c6685703864fd8bcb50adca11929d1 [id=27451a477d784e5996dd0c52eec8328a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.795755 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5f67e7b28b4240618e1817bb47d5633d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f67e7b28b4240618e1817bb47d5633d already in progress: creating tablet
I20251029 08:52:27.795977 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2916d23a0a4e42a9a143a47b047d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.792048 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_691d182612d74f709b0e5d106e4a5e11" } modify_external_catalogs: true
I20251029 08:52:27.797302 2359 raft_consensus.cc:493] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.797413 2359 raft_consensus.cc:515] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.797943 2359 leader_election.cc:290] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:27.798177 720 leader_election.cc:343] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.798280 720 leader_election.cc:304] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.798710 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9907d044daa145168417371c2bd66544" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.798659 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.799108 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 486a3fc1b6b24028b23fc8ab92832358 (DEFAULT_TABLE table=default.Table_66e4206742ae4022be3b1011f9bb49be [id=b6b631b6f4664a17889946119510dcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.799386 2458 raft_consensus.cc:2749] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.799477 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486a3fc1b6b24028b23fc8ab92832358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.799984 783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9adbcf3f12104fb5bfd1d9dd5c946c54" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:27.800268 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 56f9308235f145ee838a55434f772bfa (DEFAULT_TABLE table=default.Table_e15c46fe42ab4546913d8f009b074838 [id=5676f1f1a4c84e52a984cb3a60a8c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.800446 888 tablet_service.cc:1505] Processing CreateTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 (DEFAULT_TABLE table=default.Table_ccddc094f01348a5b25ebd260db6d8cd [id=7292f42c9fd548ecb91633f63eed6e8f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.800509 998 leader_election.cc:343] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Illegal state: Tablet not RUNNING: STOPPED
I20251029 08:52:27.800601 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f9308235f145ee838a55434f772bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.800793 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea40ed38c0ba4d2f9045188af226ced5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.800737 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ab56045bd814d59831a98618c759af4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.804230 897 tablet_service.cc:1505] Processing CreateTablet for tablet 39578ce0411646eea7f6d8dc61490e03 (DEFAULT_TABLE table=default.Table_40f4c90bb63e4ebdbf7af5f31149e57e [id=4ee6cf11f6374d3d9e0a33f2d670ee55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.804545 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39578ce0411646eea7f6d8dc61490e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.806576 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 208530f494ed41c19ab024aeab41445a (DEFAULT_TABLE table=default.Table_67c25e495c4d4292a9b38ecba68278e3 [id=c3ebbef704404eaeac9b87de9c8958a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.806890 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208530f494ed41c19ab024aeab41445a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.808012 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:27.808058 861 consensus_peers.cc:597] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c1a8a1e4cc14f1aa6e4f91174f45127. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.814699 2295 consensus_queue.cc:1048] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:27.816731 861 consensus_peers.cc:597] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa0ff14e19b4ad7a8b33796481f0f3f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.817592 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.817714 899 tablet_service.cc:1505] Processing CreateTablet for tablet 10848bd6d33e497cba4f8bb34db49c1f (DEFAULT_TABLE table=default.Table_ef2ebd3d2f9d4f03b009a7e13471cfa6 [id=22eab7d027ab46139eee251073e00fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.818071 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10848bd6d33e497cba4f8bb34db49c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.818397 757 tablet_service.cc:1505] Processing CreateTablet for tablet 10848bd6d33e497cba4f8bb34db49c1f (DEFAULT_TABLE table=default.Table_ef2ebd3d2f9d4f03b009a7e13471cfa6 [id=22eab7d027ab46139eee251073e00fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.819115 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10848bd6d33e497cba4f8bb34db49c1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.825497 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 431ab59e96274dde83ded7981b93e3d4 consensus state has no leader
I20251029 08:52:27.826637 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e687f593eead488db4bb445a63beeec7 (table default.Table_d707b992dd69414897e525d86fb4b002 [id=a23a597c8fd343769b517507845104ff]) successfully deleted
I20251029 08:52:27.826726 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:27.827286 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef67d5cbc902492fb1252229b4b7f28f consensus state has no leader
I20251029 08:52:27.827670 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_63f64af28ee546ca9bd6d0c3c75cf732"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:27.828259 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63f64af28ee546ca9bd6d0c3c75cf732 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:27.828709 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 980e7125f59140c8876e4befc1becd78
I20251029 08:52:27.829238 591 master_service.cc:954] ReplaceTablet: received request to replace tablet bdfda0866a614165824d1a3f7ad89af7 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:27.830415 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 980e7125f59140c8876e4befc1becd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:27.831002 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.831666 1044 tablet_service.cc:1505] Processing CreateTablet for tablet ec4a6972e983445bae78b31a7f1ad718 (DEFAULT_TABLE table=default.Table_2471fce4e7254548909d80a8b0c78ea9 [id=d4db2c97f8024ef6a2ac84cb470116ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.832722 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4a6972e983445bae78b31a7f1ad718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.834096 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 46583c1d493a48e398142928a5ed26ab (DEFAULT_TABLE table=default.Table_45635496357a40e4ba3c778e9ca8b7b0 [id=862c691051a249d998152925cb0a7467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.834421 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46583c1d493a48e398142928a5ed26ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.835491 1177 tablet_bootstrap.cc:654] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.842495 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:27.847551 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 10baa0547180493f872a23614cf4c608 (table default.Table_79da38cb82164ec5acce7f9f37123031 [id=56e0b03feb7245d59f0f2ef3a8c977f7]) successfully deleted
I20251029 08:52:27.849682 1050 tablet_service.cc:1505] Processing CreateTablet for tablet af3f21823277420f8f9897d3956c6a4f (DEFAULT_TABLE table=default.Table_4a5189be4b234b599b25a1f049a1b849 [id=0b388756e3c64793b18b11e52523f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.850064 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3f21823277420f8f9897d3956c6a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.851363 764 tablet_service.cc:1505] Processing CreateTablet for tablet 39578ce0411646eea7f6d8dc61490e03 (DEFAULT_TABLE table=default.Table_40f4c90bb63e4ebdbf7af5f31149e57e [id=4ee6cf11f6374d3d9e0a33f2d670ee55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.851563 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 980e7125f59140c8876e4befc1becd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:27.851979 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39578ce0411646eea7f6d8dc61490e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.852840 2707 raft_consensus.cc:493] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.852998 2707 raft_consensus.cc:515] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.853806 2707 leader_election.cc:290] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:27.854338 998 consensus_peers.cc:597] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa0ef904c2fb4dfeadbf56ae659c4e20. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.855741 905 tablet_service.cc:1505] Processing CreateTablet for tablet f2afefb80dc14001811ddb811f21fd6d (DEFAULT_TABLE table=default.Table_c5f70a8f965d4999a64b666bffc151c1 [id=7c8eae3dfd3e42a8888990aeadc118a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.856273 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fb69271e14fe3a8d0b29d0db0dc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.856642 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc95257c937454aa6cd55b5a9394a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.857208 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 0d8a73e1aad5479eb07282c642297dc0 (DEFAULT_TABLE table=default.Table_ea52c1c186c8469b9a0a7bfba7654b5d [id=157bea50919a40eb9532dbd490291bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.857550 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8a73e1aad5479eb07282c642297dc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:27.857810 998 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:27.857789 886 tablet_service.cc:1505] Processing CreateTablet for tablet 473c35faedf04353b1bf8971b268d5f3 (DEFAULT_TABLE table=default.Table_2696186101b6433885b1929577db93d7 [id=96a093af6de342efbb04624f20ff3890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.859122 2554 consensus_queue.cc:1048] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:27.865298 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 252faeca036f416493fd050d76a678f7 (DEFAULT_TABLE table=default.Table_d03243f8356b4f47b90023ee624bbb95 [id=744bd6bb788040cfa21a5905d630753d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.865818 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252faeca036f416493fd050d76a678f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.869649 596 catalog_manager.cc:5649] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:27.880457 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a94e89ac464f4105987c9db0d6a6f390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:27.881511 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb9c8c7286149128c2741635c242341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.882094 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 473c35faedf04353b1bf8971b268d5f3 (DEFAULT_TABLE table=default.Table_2696186101b6433885b1929577db93d7 [id=96a093af6de342efbb04624f20ff3890]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.882323 896 tablet_service.cc:1505] Processing CreateTablet for tablet a8104bbc33e84e8e8ceecd4e1188e034 (DEFAULT_TABLE table=default.Table_982b0112dd78419e8ea54b9a114951b9 [id=bdd8da2a2e2e464ba44871dceea7955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.882474 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473c35faedf04353b1bf8971b268d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.883078 747 tablet_service.cc:1505] Processing CreateTablet for tablet a8104bbc33e84e8e8ceecd4e1188e034 (DEFAULT_TABLE table=default.Table_982b0112dd78419e8ea54b9a114951b9 [id=bdd8da2a2e2e464ba44871dceea7955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.890369 2554 raft_consensus.cc:493] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:27.890686 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_8d966fcf45004abdba1b04d1722be5c5 [id=32fdba980db34781a7f359080cb1e56c]). Expected version 2 got 0
I20251029 08:52:27.890666 2554 raft_consensus.cc:515] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.891507 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e213713757949caa3f504f70026b7b5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:27.891659 1067 raft_consensus.cc:2468] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:27.891983 861 leader_election.cc:304] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:27.895071 861 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e213713757949caa3f504f70026b7b5
I20251029 08:52:27.896241 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3 (DEFAULT_TABLE table=default.Table_dd61164773f54438b96f450cc899ab1a [id=d5714b9aeb4648eaa0aad72b1c99d779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.896420 893 tablet_service.cc:1505] Processing CreateTablet for tablet bd952b805c9e41169f4f848657c16438 (DEFAULT_TABLE table=default.Table_b4b6fc6f5ffa4c95b59c20c1aa578af9 [id=3076f19dfd2749678b5ec9dcdc30dd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.896241 894 tablet_service.cc:1505] Processing CreateTablet for tablet 57ae0410df8d4983be1968f404bbb8c9 (DEFAULT_TABLE table=default.Table_39789940cb7e4119892e076e8929054f [id=5e8d22ff73d447979071b9ec0a9fc406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.898046 2554 leader_election.cc:290] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.898013 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3 (DEFAULT_TABLE table=default.Table_dd61164773f54438b96f450cc899ab1a [id=d5714b9aeb4648eaa0aad72b1c99d779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.898231 745 tablet_service.cc:1505] Processing CreateTablet for tablet 57ae0410df8d4983be1968f404bbb8c9 (DEFAULT_TABLE table=default.Table_39789940cb7e4119892e076e8929054f [id=5e8d22ff73d447979071b9ec0a9fc406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.898308 2554 raft_consensus.cc:2804] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:27.898370 2554 raft_consensus.cc:493] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:27.898450 2554 raft_consensus.cc:3060] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:27.899016 758 tablet_service.cc:1505] Processing CreateTablet for tablet bd952b805c9e41169f4f848657c16438 (DEFAULT_TABLE table=default.Table_b4b6fc6f5ffa4c95b59c20c1aa578af9 [id=3076f19dfd2749678b5ec9dcdc30dd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.899147 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 39578ce0411646eea7f6d8dc61490e03 (DEFAULT_TABLE table=default.Table_40f4c90bb63e4ebdbf7af5f31149e57e [id=4ee6cf11f6374d3d9e0a33f2d670ee55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.899291 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 10848bd6d33e497cba4f8bb34db49c1f (DEFAULT_TABLE table=default.Table_ef2ebd3d2f9d4f03b009a7e13471cfa6 [id=22eab7d027ab46139eee251073e00fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.899493 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39578ce0411646eea7f6d8dc61490e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.899758 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10848bd6d33e497cba4f8bb34db49c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.899786 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7abb56957a9b4661bf096bf4b4e6bbb7 (DEFAULT_TABLE table=default.Table_09a0437aa25049d8acd5bdf172b6e934 [id=c79310acc13941c9957214daf081b2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.899842 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 980e7125f59140c8876e4befc1becd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.907704 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 57ae0410df8d4983be1968f404bbb8c9 (DEFAULT_TABLE table=default.Table_39789940cb7e4119892e076e8929054f [id=5e8d22ff73d447979071b9ec0a9fc406]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:27.907835 720 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.907972 720 leader_election.cc:304] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.908241 1039 tablet_service.cc:1505] Processing CreateTablet for tablet bd952b805c9e41169f4f848657c16438 (DEFAULT_TABLE table=default.Table_b4b6fc6f5ffa4c95b59c20c1aa578af9 [id=3076f19dfd2749678b5ec9dcdc30dd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.908584 2458 raft_consensus.cc:2749] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.908840 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ae0410df8d4983be1968f404bbb8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.909149 1050 tablet_service.cc:1505] Processing CreateTablet for tablet f2afefb80dc14001811ddb811f21fd6d (DEFAULT_TABLE table=default.Table_c5f70a8f965d4999a64b666bffc151c1 [id=7c8eae3dfd3e42a8888990aeadc118a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.909535 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2afefb80dc14001811ddb811f21fd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.912153 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd952b805c9e41169f4f848657c16438. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.912528 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9adbcf3f12104fb5bfd1d9dd5c946c54" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:27.913193 720 leader_election.cc:343] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.913323 720 leader_election.cc:304] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.913781 2458 raft_consensus.cc:2749] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:27.914444 1045 tablet_service.cc:1505] Processing CreateTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 (DEFAULT_TABLE table=default.Table_ccddc094f01348a5b25ebd260db6d8cd [id=7292f42c9fd548ecb91633f63eed6e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.914829 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea40ed38c0ba4d2f9045188af226ced5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.916410 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3 (DEFAULT_TABLE table=default.Table_dd61164773f54438b96f450cc899ab1a [id=d5714b9aeb4648eaa0aad72b1c99d779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.919385 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.922559 1051 tablet_service.cc:1505] Processing CreateTablet for tablet a8104bbc33e84e8e8ceecd4e1188e034 (DEFAULT_TABLE table=default.Table_982b0112dd78419e8ea54b9a114951b9 [id=bdd8da2a2e2e464ba44871dceea7955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:27.922936 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8104bbc33e84e8e8ceecd4e1188e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.934654 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e65b3b1c8d14743aca1465d95ace95f" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:27.935048 1179 ts_tablet_manager.cc:1929] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 08:52:27.935189 1179 log.cc:1199] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a73a891fb6034f989d29a3f837303d84
I20251029 08:52:27.935606 1179 ts_tablet_manager.cc:1950] T a73a891fb6034f989d29a3f837303d84 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.935999 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 965) took 3358 ms. Trace:
I20251029 08:52:27.936164 1179 rpcz_store.cc:276] 1029 08:52:24.577535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:24.577592 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:27.720149 (+3142557us) ts_tablet_manager.cc:1151] Deleting tablet a73a891fb6034f989d29a3f837303d84
1029 08:52:27.720159 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.732101 (+ 11942us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.934966 (+202865us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.935986 (+ 1020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3142345}
I20251029 08:52:27.936671 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 966) took 2920 ms. Trace:
I20251029 08:52:27.936794 1179 rpcz_store.cc:276] 1029 08:52:25.016537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.016592 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:27.936564 (+2919972us) ts_tablet_manager.cc:1151] Deleting tablet 25bd82e3549442ada2503518037127fc
1029 08:52:27.936574 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.936656 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2919772}
I20251029 08:52:27.937093 1179 tablet_replica.cc:333] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:27.937302 1179 raft_consensus.cc:2243] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:27.937580 995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251029 08:52:27.937949 1179 raft_consensus.cc:2272] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.943212 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8104bbc33e84e8e8ceecd4e1188e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.943537 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd952b805c9e41169f4f848657c16438. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.943732 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.947254 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abb56957a9b4661bf096bf4b4e6bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.950546 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ae0410df8d4983be1968f404bbb8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:27.957681 1974 ts_tablet_manager.cc:1929] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:27.959621 1974 log.cc:1199] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/667c71a8dfd54c05ab56e845653c0c9a
I20251029 08:52:27.960419 1974 ts_tablet_manager.cc:1950] T 667c71a8dfd54c05ab56e845653c0c9a P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:27.931918 2554 raft_consensus.cc:515] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.967778 2554 leader_election.cc:290] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:27.970144 1179 ts_tablet_manager.cc:1916] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:27.970784 861 leader_election.cc:343] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e213713757949caa3f504f70026b7b5
I20251029 08:52:27.975947 1178 tablet_bootstrap.cc:654] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:27.979763 1176 ts_tablet_manager.cc:1929] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:27.979887 1176 log.cc:1199] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9adbcf3f12104fb5bfd1d9dd5c946c54
I20251029 08:52:27.982012 1176 ts_tablet_manager.cc:1950] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:27.982755 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1048) took 1741 ms. Trace:
I20251029 08:52:27.982846 1176 rpcz_store.cc:276] 1029 08:52:26.241625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.241705 (+ 80us) service_pool.cc:225] Handling call
1029 08:52:27.538550 (+1296845us) ts_tablet_manager.cc:1151] Deleting tablet 9adbcf3f12104fb5bfd1d9dd5c946c54
1029 08:52:27.538559 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.755341 (+216782us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.953877 (+198536us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.982735 (+ 28858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296637}
I20251029 08:52:27.983167 1176 tablet_replica.cc:333] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:27.983355 1176 raft_consensus.cc:2243] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:27.983500 1176 raft_consensus.cc:2272] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:27.967343 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e213713757949caa3f504f70026b7b5" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:27.985813 1067 raft_consensus.cc:3060] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:27.980367 1183 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:27.987658 1183 ts_tablet_manager.cc:1403] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.245s user 0.024s sys 0.060s
I20251029 08:52:27.988407 1183 raft_consensus.cc:359] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.988565 1183 raft_consensus.cc:385] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:27.988606 1183 raft_consensus.cc:740] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:27.988816 1183 consensus_queue.cc:260] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:27.988996 1067 raft_consensus.cc:2468] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:27.952327 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 21d409c2e24947beb99351da46a5aef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:27.989416 861 leader_election.cc:304] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:27.989858 2731 raft_consensus.cc:2804] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:27.990022 2731 raft_consensus.cc:697] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:27.990284 2731 consensus_queue.cc:237] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:27.992815 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fc8d5092b4774b28adcf5f8a06dca48d" } modify_external_catalogs: true
W20251029 08:52:27.993731 996 consensus_peers.cc:597] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:27.994920 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fc8d5092b4774b28adcf5f8a06dca48d" } modify_external_catalogs: true
W20251029 08:52:27.994546 996 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.997648 996 leader_election.cc:304] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.998057 2359 raft_consensus.cc:2749] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:27.998376 996 leader_election.cc:343] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.998626 996 leader_election.cc:304] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:27.998971 2359 raft_consensus.cc:2749] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:27.999430 996 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:27.999661 996 leader_election.cc:304] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.000000 2359 raft_consensus.cc:2749] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.002547 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e15c46fe42ab4546913d8f009b074838" } modify_external_catalogs: true
I20251029 08:52:28.002754 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e15c46fe42ab4546913d8f009b074838" } modify_external_catalogs: true
W20251029 08:52:28.005095 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94e89ac464f4105987c9db0d6a6f390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.006075 1178 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:28.006306 1178 ts_tablet_manager.cc:1403] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.283s user 0.010s sys 0.004s
I20251029 08:52:28.006603 901 tablet_service.cc:1505] Processing CreateTablet for tablet 7abb56957a9b4661bf096bf4b4e6bbb7 (DEFAULT_TABLE table=default.Table_09a0437aa25049d8acd5bdf172b6e934 [id=c79310acc13941c9957214daf081b2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.007014 898 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb886dc40c54adfba78ed85bd950b13 (DEFAULT_TABLE table=default.Table_9b4e0264c9d14e7cb206bd0bafa0a120 [id=ae08afc43f3849f58702ae04be1eadd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.007388 1178 raft_consensus.cc:359] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.007651 1178 raft_consensus.cc:385] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.007764 1178 raft_consensus.cc:740] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.008003 1178 consensus_queue.cc:260] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.008249 895 tablet_service.cc:1505] Processing CreateTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b (DEFAULT_TABLE table=default.Table_01f74d8d149e47da972673a83c50e16c [id=085d92c1485140aaac0e96186652ae8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.008548 1178 ts_tablet_manager.cc:1434] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:28.008800 1178 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:28.010916 589 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 6a7091ff6bed4e3a8447d9de8ea9fb48 (table default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df]). Expected version 1 got 0
I20251029 08:52:28.013662 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_af67eaef7a72468c86b5c55ccb6d2626"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:28.021756 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e213713757949caa3f504f70026b7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.024070 2451 raft_consensus.cc:493] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.024302 2451 raft_consensus.cc:515] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.024513 1179 ts_tablet_manager.cc:1929] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.025357 1177 tablet_bootstrap.cc:492] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:28.025574 1177 ts_tablet_manager.cc:1403] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.305s user 0.009s sys 0.009s
I20251029 08:52:28.025892 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1004) took 2997 ms. Trace:
I20251029 08:52:28.026028 1974 rpcz_store.cc:276] 1029 08:52:25.028546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.028599 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:27.733865 (+2705266us) ts_tablet_manager.cc:1151] Deleting tablet 667c71a8dfd54c05ab56e845653c0c9a
1029 08:52:27.733876 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.890260 (+156384us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:27.957597 (+ 67337us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.025874 (+ 68277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705095}
I20251029 08:52:28.026360 1177 raft_consensus.cc:359] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:28.026436 861 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24834985a71349b9865840e9b5537d72
I20251029 08:52:28.026564 2451 leader_election.cc:290] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.026671 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1008) took 2968 ms. Trace:
I20251029 08:52:28.026818 1974 rpcz_store.cc:276] 1029 08:52:25.057731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.057791 (+ 60us) service_pool.cc:225] Handling call
1029 08:52:28.026565 (+2968774us) ts_tablet_manager.cc:1151] Deleting tablet 930314c8f90f462990f93fab8f7790c2
1029 08:52:28.026574 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.026654 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2961639}
I20251029 08:52:28.027319 1974 tablet_replica.cc:333] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:28.027570 1974 raft_consensus.cc:2243] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:28.027915 1178 tablet_bootstrap.cc:654] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.027937 1974 raft_consensus.cc:2272] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.028545 1974 ts_tablet_manager.cc:1916] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.026705 1177 raft_consensus.cc:385] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.028745 1177 raft_consensus.cc:740] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.029048 1177 consensus_queue.cc:260] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.029645 1177 ts_tablet_manager.cc:1434] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251029 08:52:28.034345 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af67eaef7a72468c86b5c55ccb6d2626 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.025426 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24834985a71349b9865840e9b5537d72" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.034552 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 667c71a8dfd54c05ab56e845653c0c9a (table default.Table_c9c0266a3d25476cae75216163270a5f [id=3b0c410e5d1e4d33ab2f50cff45fb590]) successfully deleted
I20251029 08:52:28.034683 1067 raft_consensus.cc:2468] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:28.034972 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 930314c8f90f462990f93fab8f7790c2 because the tablet was not found. No further retry: Not found: Tablet not found: 930314c8f90f462990f93fab8f7790c2
I20251029 08:52:28.035112 861 leader_election.cc:304] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.035595 2451 raft_consensus.cc:2804] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.035719 2451 raft_consensus.cc:493] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.035835 2451 raft_consensus.cc:3060] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.040716 1179 log.cc:1199] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/667c71a8dfd54c05ab56e845653c0c9a
I20251029 08:52:28.041203 1179 ts_tablet_manager.cc:1950] T 667c71a8dfd54c05ab56e845653c0c9a P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.042033 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 968) took 3009 ms. Trace:
I20251029 08:52:28.042301 2451 raft_consensus.cc:515] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.042627 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3fffb85f46a84893b72083c6894e6f84" } modify_external_catalogs: true
I20251029 08:52:28.044090 888 tablet_service.cc:1505] Processing CreateTablet for tablet 7d93e6ba0e5b4989a33e4ae7ad757046 (DEFAULT_TABLE table=default.Table_f117335b909e430dab2a5a5da90c9434 [id=4fba0300a8be408c8c3e656c3d32d61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.044879 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3fffb85f46a84893b72083c6894e6f84" } modify_external_catalogs: true
I20251029 08:52:28.044981 1179 rpcz_store.cc:276] 1029 08:52:25.032236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.032282 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:27.937057 (+2904775us) ts_tablet_manager.cc:1151] Deleting tablet 667c71a8dfd54c05ab56e845653c0c9a
1029 08:52:27.937065 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:27.997848 (+ 60783us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.024389 (+ 26541us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.042013 (+ 17624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2904540}
I20251029 08:52:28.045365 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:28.045392 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 967) took 3013 ms. Trace:
I20251029 08:52:28.045475 1179 rpcz_store.cc:276] 1029 08:52:25.032130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.032920 (+ 790us) service_pool.cc:225] Handling call
1029 08:52:28.045263 (+3012343us) ts_tablet_manager.cc:1151] Deleting tablet 5f67e7b28b4240618e1817bb47d5633d
1029 08:52:28.045274 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.045376 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3012131}
I20251029 08:52:28.045763 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 972) took 2903 ms. Trace:
W20251029 08:52:28.047454 861 leader_election.cc:343] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 24834985a71349b9865840e9b5537d72
I20251029 08:52:28.048632 897 tablet_service.cc:1505] Processing CreateTablet for tablet c91d6f5a73e248c8b2fec59abc8e56b2 (DEFAULT_TABLE table=default.Table_9830b3c1094348f59df346203a20a135 [id=f2220e971bde45d182f9fa5c4d8f351d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.049062 899 tablet_service.cc:1505] Processing CreateTablet for tablet 957d0d9c57e34e9eaf1f79da53a33c6f (DEFAULT_TABLE table=default.Table_9cb7ad5e75534b2897602246ecb55a74 [id=b4314fb513754af48306a1736282d6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.049515 2451 leader_election.cc:290] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.032328 2359 raft_consensus.cc:493] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.049820 2359 raft_consensus.cc:515] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.050382 2359 leader_election.cc:290] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.046463 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24834985a71349b9865840e9b5537d72" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:28.050591 1067 raft_consensus.cc:3060] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.052374 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50e2fdfc0e64516a41c90f5a56f4c89" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.052567 922 raft_consensus.cc:2468] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:28.053498 1067 raft_consensus.cc:2468] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:28.054342 861 leader_election.cc:304] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.055270 2451 raft_consensus.cc:2804] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:28.055442 2451 raft_consensus.cc:697] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:28.055712 2451 consensus_queue.cc:237] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.055932 1179 rpcz_store.cc:276] 1029 08:52:25.142285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.142553 (+ 268us) service_pool.cc:225] Handling call
1029 08:52:28.045673 (+2903120us) ts_tablet_manager.cc:1151] Deleting tablet a2c13db40d854269b13340aa6acef83e
1029 08:52:28.045682 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.045751 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902882}
I20251029 08:52:28.056581 1178 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:28.056795 1178 ts_tablet_manager.cc:1403] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.003s
I20251029 08:52:28.056779 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 973) took 2914 ms. Trace:
I20251029 08:52:28.056854 1179 rpcz_store.cc:276] 1029 08:52:25.142479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.142837 (+ 358us) service_pool.cc:225] Handling call
1029 08:52:28.056643 (+2913806us) ts_tablet_manager.cc:1151] Deleting tablet 8b1af4bcd7af49a0b8a419298ab37ea9
1029 08:52:28.056656 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.056763 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2913694}
I20251029 08:52:28.054961 996 leader_election.cc:304] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:28.057547 998 leader_election.cc:343] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:28.058079 2359 raft_consensus.cc:2804] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.058212 2359 raft_consensus.cc:493] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.058342 2359 raft_consensus.cc:3060] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.060073 1178 raft_consensus.cc:359] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.060344 1178 raft_consensus.cc:385] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.060436 1178 raft_consensus.cc:740] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.060729 1178 consensus_queue.cc:260] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.061882 2359 raft_consensus.cc:515] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.062639 2359 leader_election.cc:290] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.064150 1178 ts_tablet_manager.cc:1434] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:28.065302 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 978) took 2845 ms. Trace:
I20251029 08:52:28.065232 752 tablet_service.cc:1505] Processing CreateTablet for tablet 7d93e6ba0e5b4989a33e4ae7ad757046 (DEFAULT_TABLE table=default.Table_f117335b909e430dab2a5a5da90c9434 [id=4fba0300a8be408c8c3e656c3d32d61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.065397 758 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.065375 1179 rpcz_store.cc:276] 1029 08:52:25.219566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.219741 (+ 175us) service_pool.cc:225] Handling call
1029 08:52:28.057067 (+2837326us) ts_tablet_manager.cc:1151] Deleting tablet f062556d67a148d19d242b8d99cfdff9
1029 08:52:28.057077 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.065285 (+ 8208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837163}
I20251029 08:52:28.066114 764 tablet_service.cc:1505] Processing CreateTablet for tablet c91d6f5a73e248c8b2fec59abc8e56b2 (DEFAULT_TABLE table=default.Table_9830b3c1094348f59df346203a20a135 [id=f2220e971bde45d182f9fa5c4d8f351d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.066166 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 7abb56957a9b4661bf096bf4b4e6bbb7 (DEFAULT_TABLE table=default.Table_09a0437aa25049d8acd5bdf172b6e934 [id=c79310acc13941c9957214daf081b2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.066748 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b (DEFAULT_TABLE table=default.Table_01f74d8d149e47da972673a83c50e16c [id=085d92c1485140aaac0e96186652ae8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.066767 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50e2fdfc0e64516a41c90f5a56f4c89" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:28.066960 922 raft_consensus.cc:3060] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.067296 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb886dc40c54adfba78ed85bd950b13 (DEFAULT_TABLE table=default.Table_9b4e0264c9d14e7cb206bd0bafa0a120 [id=ae08afc43f3849f58702ae04be1eadd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.067555 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abb56957a9b4661bf096bf4b4e6bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.067667 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 7d93e6ba0e5b4989a33e4ae7ad757046 (DEFAULT_TABLE table=default.Table_f117335b909e430dab2a5a5da90c9434 [id=4fba0300a8be408c8c3e656c3d32d61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.068243 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dec9fdf1ae94ae687f30f89096ce89b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:28.068289 998 leader_election.cc:343] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:28.068893 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93e6ba0e5b4989a33e4ae7ad757046. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.070181 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d6f5a73e248c8b2fec59abc8e56b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.075181 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb886dc40c54adfba78ed85bd950b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.076117 1178 tablet_bootstrap.cc:492] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:28.077813 1178 tablet_bootstrap.cc:654] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.079944 1179 tablet_replica.cc:333] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.080387 1179 raft_consensus.cc:2243] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.081131 1179 raft_consensus.cc:2272] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.081833 1179 ts_tablet_manager.cc:1916] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.085214 922 raft_consensus.cc:2468] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:28.085843 996 leader_election.cc:304] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.086149 2359 raft_consensus.cc:2804] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:28.086689 2359 raft_consensus.cc:697] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:28.086983 2359 consensus_queue.cc:237] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.097556 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a73a891fb6034f989d29a3f837303d84 (table default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]) successfully deleted
W20251029 08:52:28.097929 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 25bd82e3549442ada2503518037127fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bd82e3549442ada2503518037127fc already in progress: creating tablet
I20251029 08:52:28.104429 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb886dc40c54adfba78ed85bd950b13 (DEFAULT_TABLE table=default.Table_9b4e0264c9d14e7cb206bd0bafa0a120 [id=ae08afc43f3849f58702ae04be1eadd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.105193 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb886dc40c54adfba78ed85bd950b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.114369 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 420c3357ebff4dd08df4ed2477b40ddf
I20251029 08:52:28.116072 2458 raft_consensus.cc:493] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.116222 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:28.116210 2458 raft_consensus.cc:515] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.116509 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 667c71a8dfd54c05ab56e845653c0c9a (table default.Table_c9c0266a3d25476cae75216163270a5f [id=3b0c410e5d1e4d33ab2f50cff45fb590]) successfully deleted
I20251029 08:52:28.116556 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66eb18b4419b42f087e8468a9dc15153" } new_table_name: "default.Table_33c34c28c5aa4dcf8d5408580a8a61f4" modify_external_catalogs: true
W20251029 08:52:28.116809 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5f67e7b28b4240618e1817bb47d5633d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f67e7b28b4240618e1817bb47d5633d already in progress: creating tablet
I20251029 08:52:28.116892 2458 leader_election.cc:290] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:28.117089 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a2c13db40d854269b13340aa6acef83e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c13db40d854269b13340aa6acef83e already in progress: creating tablet
I20251029 08:52:28.117821 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.118257 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:28.118320 720 leader_election.cc:343] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:28.119035 718 leader_election.cc:343] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.119149 718 leader_election.cc:304] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.119385 2458 raft_consensus.cc:2749] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.120103 745 tablet_service.cc:1505] Processing CreateTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b (DEFAULT_TABLE table=default.Table_01f74d8d149e47da972673a83c50e16c [id=085d92c1485140aaac0e96186652ae8a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:28.122655 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b1af4bcd7af49a0b8a419298ab37ea9 already in progress: creating tablet
W20251029 08:52:28.122974 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f062556d67a148d19d242b8d99cfdff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f062556d67a148d19d242b8d99cfdff9 already in progress: creating tablet
W20251029 08:52:28.123713 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bdfda0866a614165824d1a3f7ad89af7 (table default.Table_6fe654db95e54d37a885e5e31bc35b37 [id=c488f3afc154469cab70bf8007773b8f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bdfda0866a614165824d1a3f7ad89af7 consensus state has no leader
I20251029 08:52:28.124366 752 tablet_service.cc:1505] Processing CreateTablet for tablet 957d0d9c57e34e9eaf1f79da53a33c6f (DEFAULT_TABLE table=default.Table_9cb7ad5e75534b2897602246ecb55a74 [id=b4314fb513754af48306a1736282d6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.124339 1051 tablet_service.cc:1505] Processing CreateTablet for tablet c91d6f5a73e248c8b2fec59abc8e56b2 (DEFAULT_TABLE table=default.Table_9830b3c1094348f59df346203a20a135 [id=f2220e971bde45d182f9fa5c4d8f351d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.124953 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dec9fdf1ae94ae687f30f89096ce89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.120692 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]). Expected version 1 got 0
I20251029 08:52:28.127861 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_28cea2a6590a436e98fce31e8bf4eff3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:28.128499 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28cea2a6590a436e98fce31e8bf4eff3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.129374 2451 raft_consensus.cc:493] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.129510 2451 raft_consensus.cc:515] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.130223 2451 leader_election.cc:290] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:28.131453 861 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:28.127419 1178 tablet_bootstrap.cc:492] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:28.136950 1178 ts_tablet_manager.cc:1403] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.000s
I20251029 08:52:28.137871 1178 raft_consensus.cc:359] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.138067 1178 raft_consensus.cc:385] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.138121 1178 raft_consensus.cc:740] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.138350 1178 consensus_queue.cc:260] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.138927 1178 ts_tablet_manager.cc:1434] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:28.139235 1178 tablet_bootstrap.cc:492] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:28.139398 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d6f5a73e248c8b2fec59abc8e56b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.139395 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bbf6401423411089c9070f3b03f1a7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.139676 1067 raft_consensus.cc:2468] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:28.140172 861 leader_election.cc:304] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.140430 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957d0d9c57e34e9eaf1f79da53a33c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.141651 2451 raft_consensus.cc:2804] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.141741 2451 raft_consensus.cc:493] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.141796 2451 raft_consensus.cc:3060] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.144411 1179 ts_tablet_manager.cc:1929] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.144620 1179 log.cc:1199] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/39fc9612689b46d498d6092bd552da4e
I20251029 08:52:28.145009 1179 ts_tablet_manager.cc:1950] T 39fc9612689b46d498d6092bd552da4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.147579 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 979) took 2845 ms. Trace:
I20251029 08:52:28.148738 2451 raft_consensus.cc:515] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.149646 1179 rpcz_store.cc:276] 1029 08:52:25.302037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.302096 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:28.065624 (+2763528us) ts_tablet_manager.cc:1151] Deleting tablet 39fc9612689b46d498d6092bd552da4e
1029 08:52:28.065634 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.115291 (+ 49657us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.144298 (+ 29007us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.147532 (+ 3234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2763219}
I20251029 08:52:28.150090 1179 tablet_replica.cc:333] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.150122 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bbf6401423411089c9070f3b03f1a7" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:28.150279 1067 raft_consensus.cc:3060] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.150347 1179 raft_consensus.cc:2243] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.150508 1179 raft_consensus.cc:2272] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.151010 1179 ts_tablet_manager.cc:1916] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.151252 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 39fc9612689b46d498d6092bd552da4e (table default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]) successfully deleted
I20251029 08:52:28.152863 1067 raft_consensus.cc:2468] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:28.155328 861 leader_election.cc:304] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:28.155614 2731 raft_consensus.cc:2804] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:28.155735 2731 raft_consensus.cc:697] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:28.155973 2731 consensus_queue.cc:237] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.158636 1178 tablet_bootstrap.cc:654] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.159454 2451 leader_election.cc:290] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:28.160185 861 leader_election.cc:343] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:28.178869 1179 ts_tablet_manager.cc:1929] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.179286 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93e6ba0e5b4989a33e4ae7ad757046. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.186026 1179 log.cc:1199] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/652903cbbdb042d780e7d71fc2c0ab4d
I20251029 08:52:28.186502 1179 ts_tablet_manager.cc:1950] T 652903cbbdb042d780e7d71fc2c0ab4d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.189543 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 981) took 2887 ms. Trace:
I20251029 08:52:28.189718 1179 rpcz_store.cc:276] 1029 08:52:25.302327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.302369 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:28.150041 (+2847672us) ts_tablet_manager.cc:1151] Deleting tablet 652903cbbdb042d780e7d71fc2c0ab4d
1029 08:52:28.150051 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.168846 (+ 18795us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.178757 (+ 9911us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.189521 (+ 10764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2847433}
I20251029 08:52:28.190265 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 652903cbbdb042d780e7d71fc2c0ab4d (table default.Table_ca344fa2220d4ea09a867ad42d7fe215 [id=0e0c97d7ced7490b86425ea2e2e4530c]) successfully deleted
I20251029 08:52:28.190649 1179 tablet_replica.cc:333] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.190855 1179 raft_consensus.cc:2243] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.191067 1179 raft_consensus.cc:2272] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.191653 1179 ts_tablet_manager.cc:1916] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.202739 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 56f9308235f145ee838a55434f772bfa
I20251029 08:52:28.204269 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 56f9308235f145ee838a55434f772bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.204532 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 56f9308235f145ee838a55434f772bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.205008 1179 ts_tablet_manager.cc:1929] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.204979 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5d3db711683243eabfce1d1543d63f9f" } new_table_name: "default.Table_20173eef43ed4e66acd14cb73a7d2ebd" modify_external_catalogs: true
I20251029 08:52:28.210100 902 tablet_service.cc:1552] Processing DeleteTablet for tablet 56f9308235f145ee838a55434f772bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.210407 1179 log.cc:1199] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:28.210772 1179 ts_tablet_manager.cc:1950] T aa411391c63146e7acb8b4c6edbe12a0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.214675 2458 raft_consensus.cc:493] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:28.214824 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22f9c51cd2ec4689ba2a4d8d466d6716 consensus state has no leader
I20251029 08:52:28.214820 2458 raft_consensus.cc:515] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.215425 2458 leader_election.cc:290] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.215348 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_afe3884a169642a99f2dbbeb57bb8d9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:28.216745 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5ec3d23d22464070b4557777a2306288" } new_table_name: "default.Table_d07c3391755f4f82bd1579cd0d3b2c49" modify_external_catalogs: true
I20251029 08:52:28.216315 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:28.221084 720 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.221923 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_836882de6c804a5c908824f31f0aa328" } modify_external_catalogs: true
I20251029 08:52:28.222038 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 982) took 2919 ms. Trace:
I20251029 08:52:28.222179 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_836882de6c804a5c908824f31f0aa328" } modify_external_catalogs: true
I20251029 08:52:28.222250 1179 rpcz_store.cc:276] 1029 08:52:25.302407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.302673 (+ 266us) service_pool.cc:225] Handling call
1029 08:52:28.190598 (+2887925us) ts_tablet_manager.cc:1151] Deleting tablet aa411391c63146e7acb8b4c6edbe12a0
1029 08:52:28.190608 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.201726 (+ 11118us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.204891 (+ 3165us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.222015 (+ 17124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887691}
I20251029 08:52:28.222951 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aa411391c63146e7acb8b4c6edbe12a0 (table default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]) successfully deleted
I20251029 08:52:28.223488 1179 tablet_replica.cc:333] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.223694 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.223800 1179 raft_consensus.cc:2243] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:28.224478 718 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.224604 718 leader_election.cc:304] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.224869 2458 raft_consensus.cc:2749] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.226866 1179 raft_consensus.cc:2272] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.227548 1179 ts_tablet_manager.cc:1916] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:28.232530 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7af99e95e334279ac160ca7e0b85e77 consensus state has no leader
I20251029 08:52:28.232987 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ca344fa2220d4ea09a867ad42d7fe215" } new_table_name: "default.Table_35877e0584a84248a87841ffcb0cbf94" modify_external_catalogs: true
I20251029 08:52:28.234725 1179 ts_tablet_manager.cc:1929] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.237176 1179 log.cc:1199] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bd0c43367a8c47f390cc61d2764d61e4
I20251029 08:52:28.237675 1179 ts_tablet_manager.cc:1950] T bd0c43367a8c47f390cc61d2764d61e4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:28.238062 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adba5919a81348bb94569ca52fa4bded consensus state has no leader
I20251029 08:52:28.238523 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 980) took 2936 ms. Trace:
I20251029 08:52:28.238572 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_532d58c219f94321bf5460c99c389e1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:28.097690 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet bdfda0866a614165824d1a3f7ad89af7
I20251029 08:52:28.240675 591 catalog_manager.cc:6719] ReplaceTablet: tablet bdfda0866a614165824d1a3f7ad89af7 deleted and replaced by tablet 73db194649ac4d8caea12c56c67f78d0
I20251029 08:52:28.241058 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b6d28fc5fdc14b10b3436923d1b86bc1" } modify_external_catalogs: true
I20251029 08:52:28.241252 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b6d28fc5fdc14b10b3436923d1b86bc1" } modify_external_catalogs: true
I20251029 08:52:28.242203 1179 rpcz_store.cc:276] 1029 08:52:25.302197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.303064 (+ 867us) service_pool.cc:225] Handling call
1029 08:52:28.223403 (+2920339us) ts_tablet_manager.cc:1151] Deleting tablet bd0c43367a8c47f390cc61d2764d61e4
1029 08:52:28.223430 (+ 27us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.231302 (+ 7872us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.234619 (+ 3317us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.238504 (+ 3885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2919991}
I20251029 08:52:28.242640 1179 tablet_replica.cc:333] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.242862 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bd0c43367a8c47f390cc61d2764d61e4 (table default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]) successfully deleted
I20251029 08:52:28.243121 2359 consensus_queue.cc:1048] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:28.243131 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfda0866a614165824d1a3f7ad89af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db194649ac4d8caea12c56c67f78d0) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.243654 1179 raft_consensus.cc:2243] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.243860 1179 raft_consensus.cc:2272] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.243839 752 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfda0866a614165824d1a3f7ad89af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db194649ac4d8caea12c56c67f78d0) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.244572 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_8d966fcf45004abdba1b04d1722be5c5" } new_table_name: "default.Table_afa011ccbb98416495ed91cfb02d50fb" modify_external_catalogs: true
I20251029 08:52:28.246490 1179 ts_tablet_manager.cc:1916] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.246723 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.246754 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.250108 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 420c3357ebff4dd08df4ed2477b40ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.250382 902 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfda0866a614165824d1a3f7ad89af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db194649ac4d8caea12c56c67f78d0) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.251843 902 tablet_service.cc:1552] Processing DeleteTablet for tablet d3955ff79faf42bb82f764aae79b85d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.252568 902 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bbf6401423411089c9070f3b03f1a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 739f3ef68dc141af90954bf0da4c5d2e) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.253211 902 tablet_service.cc:1552] Processing DeleteTablet for tablet d50e2fdfc0e64516a41c90f5a56f4c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:28.254041 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afe3884a169642a99f2dbbeb57bb8d9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.256130 1179 ts_tablet_manager.cc:1929] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 08:52:28.256690 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_532d58c219f94321bf5460c99c389e1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.264693 1179 log.cc:1199] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:28.265220 1179 ts_tablet_manager.cc:1950] T 7be4bcf9e3324bd884e946fae20648fd P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.266332 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 983) took 2963 ms. Trace:
I20251029 08:52:28.266566 1179 rpcz_store.cc:276] 1029 08:52:25.302531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.305276 (+ 2745us) service_pool.cc:225] Handling call
1029 08:52:28.242587 (+2937311us) ts_tablet_manager.cc:1151] Deleting tablet 7be4bcf9e3324bd884e946fae20648fd
1029 08:52:28.242598 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.252688 (+ 10090us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.255979 (+ 3291us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.266302 (+ 10323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936948}
I20251029 08:52:28.267264 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7be4bcf9e3324bd884e946fae20648fd (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
I20251029 08:52:28.267362 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 989) took 2830 ms. Trace:
I20251029 08:52:28.267576 1179 rpcz_store.cc:276] 1029 08:52:25.437214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.437514 (+ 300us) service_pool.cc:225] Handling call
1029 08:52:28.267216 (+2829702us) ts_tablet_manager.cc:1151] Deleting tablet d0d00a6b772d4034ac7fa764884c7081
1029 08:52:28.267227 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.267342 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2829463}
W20251029 08:52:28.268253 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d0d00a6b772d4034ac7fa764884c7081 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0d00a6b772d4034ac7fa764884c7081 already in progress: creating tablet
I20251029 08:52:28.268643 1179 tablet_replica.cc:333] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.268937 1179 raft_consensus.cc:2243] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.269203 1179 raft_consensus.cc:2272] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.269829 1179 ts_tablet_manager.cc:1916] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.274947 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c5d9527d294c432db2ccf60af84731b0
I20251029 08:52:28.275807 752 tablet_service.cc:1552] Processing DeleteTablet for tablet c5d9527d294c432db2ccf60af84731b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.275892 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ecbebd3f5d9b42218bb0dd513c106df2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:28.276593 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecbebd3f5d9b42218bb0dd513c106df2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.277841 1179 ts_tablet_manager.cc:1929] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:28.277945 1179 log.cc:1199] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:28.278021 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5e35d40b730a4a7883c324a6cc93b161
I20251029 08:52:28.278280 1179 ts_tablet_manager.cc:1950] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.278805 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_36bb0544db8d43f88d006f794682fab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:28.279053 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 992) took 2808 ms. Trace:
I20251029 08:52:28.279130 1179 rpcz_store.cc:276] 1029 08:52:25.471009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.472667 (+ 1658us) service_pool.cc:225] Handling call
1029 08:52:28.268593 (+2795926us) ts_tablet_manager.cc:1151] Deleting tablet 61b07eb4befa41f3a1b5f2e9ebc2277c
1029 08:52:28.268602 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.274220 (+ 5618us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.277732 (+ 3512us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.279031 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795675}
W20251029 08:52:28.279428 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36bb0544db8d43f88d006f794682fab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.283179 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]) successfully deleted
I20251029 08:52:28.283360 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.283717 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet c5d9527d294c432db2ccf60af84731b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.283936 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.284984 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e35d40b730a4a7883c324a6cc93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.285360 902 tablet_service.cc:1552] Processing DeleteTablet for tablet c5d9527d294c432db2ccf60af84731b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.284628 1176 ts_tablet_manager.cc:1916] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.285961 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_95d0e45bf0384a5c93ef2b9a12f8275b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:28.286283 1179 tablet_replica.cc:333] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.286579 1179 raft_consensus.cc:2243] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:28.286816 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95d0e45bf0384a5c93ef2b9a12f8275b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.287746 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473c35faedf04353b1bf8971b268d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.287989 1179 raft_consensus.cc:2272] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.288888 1179 ts_tablet_manager.cc:1916] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.289927 2458 raft_consensus.cc:493] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.290103 2458 raft_consensus.cc:515] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.290869 2458 leader_election.cc:290] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.291505 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dec9fdf1ae94ae687f30f89096ce89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.291581 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:28.292171 720 leader_election.cc:343] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.292443 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:28.293031 718 leader_election.cc:343] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.293162 718 leader_election.cc:304] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.293457 2458 raft_consensus.cc:2749] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.296978 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8104bbc33e84e8e8ceecd4e1188e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.297379 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb886dc40c54adfba78ed85bd950b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.301232 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2afefb80dc14001811ddb811f21fd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.301488 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd952b805c9e41169f4f848657c16438. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.301731 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ae0410df8d4983be1968f404bbb8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.304253 2458 raft_consensus.cc:493] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.304411 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd3e1dcd8d04e13b990a8f064f69ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.304430 2458 raft_consensus.cc:515] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.305099 2458 leader_election.cc:290] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.305910 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "663d6587a5e14519afbb0f94d691d1b4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:28.306512 720 leader_election.cc:343] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.308007 2707 consensus_queue.cc:1048] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:28.308138 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "663d6587a5e14519afbb0f94d691d1b4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.308589 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abb56957a9b4661bf096bf4b4e6bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:28.309088 718 leader_election.cc:343] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.309213 718 leader_election.cc:304] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.309477 2458 raft_consensus.cc:2749] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.310736 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:28.315805 1179 ts_tablet_manager.cc:1929] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.321333 1179 log.cc:1199] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:28.323868 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap replaying log segment 1/1 (157.2K/185.7K this segment, stats: ops{read=799 overwritten=0 applied=798 ignored=0} inserts{seen=746 ignored=0} mutations{seen=908 ignored=0} orphaned_commits=0)
I20251029 08:52:28.325148 1974 ts_tablet_manager.cc:1929] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.326449 1974 log.cc:1199] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bd0c43367a8c47f390cc61d2764d61e4
I20251029 08:52:28.329586 1179 ts_tablet_manager.cc:1950] T e8c4d793f10147fba5e4f44474575bf0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:27.989316 1183 ts_tablet_manager.cc:1434] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:28.332864 1183 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:28.333967 2742 raft_consensus.cc:493] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.334228 2742 raft_consensus.cc:515] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.334360 1183 tablet_bootstrap.cc:654] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.335314 2742 leader_election.cc:290] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.335546 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f2b34642704cd6bb966b6dcd494d51" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:28.336069 720 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.336385 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f2b34642704cd6bb966b6dcd494d51" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.337625 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 957d0d9c57e34e9eaf1f79da53a33c6f (DEFAULT_TABLE table=default.Table_9cb7ad5e75534b2897602246ecb55a74 [id=b4314fb513754af48306a1736282d6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.338213 1177 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:28.339003 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93e6ba0e5b4989a33e4ae7ad757046. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:28.339047 718 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.339195 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957d0d9c57e34e9eaf1f79da53a33c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.339162 718 leader_election.cc:304] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.339409 2742 raft_consensus.cc:2749] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.339627 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d6f5a73e248c8b2fec59abc8e56b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.337090 1974 ts_tablet_manager.cc:1950] T bd0c43367a8c47f390cc61d2764d61e4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:28.341171 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 996) took 2681 ms. Trace:
I20251029 08:52:28.341279 1179 rpcz_store.cc:276] 1029 08:52:25.659880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.659929 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:28.286218 (+2626289us) ts_tablet_manager.cc:1151] Deleting tablet e8c4d793f10147fba5e4f44474575bf0
1029 08:52:28.286231 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.312682 (+ 26451us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.315719 (+ 3037us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.341152 (+ 25433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2626040}
I20251029 08:52:28.341688 752 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5eee4655b9445cb2c05c3a0307cee2 (DEFAULT_TABLE table=default.Table_980d17e6af4a4439b1f459070584dcec [id=595c8cb5dac04a1bb7f151832c8e702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.341727 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e8c4d793f10147fba5e4f44474575bf0 (table default.Table_02acde1d2bfe41e9a0f7565ed487c1c7 [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:28.342167 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5eee4655b9445cb2c05c3a0307cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.342572 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1005) took 3285 ms. Trace:
I20251029 08:52:28.346498 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a85fb70d0d2b44f6b4912645f18e8150"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:28.347097 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a85fb70d0d2b44f6b4912645f18e8150 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.349202 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9] alter complete (version 1)
I20251029 08:52:28.349917 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2cd17906960e40f59deaa3646d9f9359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:28.350467 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd17906960e40f59deaa3646d9f9359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.350198 1974 rpcz_store.cc:276] 1029 08:52:25.057058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.058690 (+ 1632us) service_pool.cc:225] Handling call
1029 08:52:28.027278 (+2968588us) ts_tablet_manager.cc:1151] Deleting tablet bd0c43367a8c47f390cc61d2764d61e4
1029 08:52:28.027286 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.290395 (+263109us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.324340 (+ 33945us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.342554 (+ 18214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2962283}
I20251029 08:52:28.353296 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.358177 1974 tablet_replica.cc:333] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:28.360050 1179 tablet_replica.cc:333] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.360216 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957d0d9c57e34e9eaf1f79da53a33c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.360298 1179 raft_consensus.cc:2243] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:28.360670 1179 raft_consensus.cc:2272] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.361408 1179 ts_tablet_manager.cc:1916] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.362516 589 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]). Expected version 1 got 0
I20251029 08:52:28.367997 2451 raft_consensus.cc:493] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.368168 2451 raft_consensus.cc:515] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.368721 1974 raft_consensus.cc:2243] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:28.369848 1179 ts_tablet_manager.cc:1929] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.370034 1179 log.cc:1199] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/43c3074caee74069a8a7f1b87721d726
I20251029 08:52:28.370541 1179 ts_tablet_manager.cc:1950] T 43c3074caee74069a8a7f1b87721d726 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.372325 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.371366 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 997) took 2710 ms. Trace:
I20251029 08:52:28.373242 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bd0c43367a8c47f390cc61d2764d61e4 (table default.Table_1e96e0acdb10410a89f8f34e1f44deff [id=0256260d3810406791442370b6681b7a]) successfully deleted
I20251029 08:52:28.373550 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.373448 1179 rpcz_store.cc:276] 1029 08:52:25.661001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.661044 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:28.342563 (+2681519us) ts_tablet_manager.cc:1151] Deleting tablet 43c3074caee74069a8a7f1b87721d726
1029 08:52:28.359977 (+ 17414us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.364871 (+ 4894us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.369739 (+ 4868us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.371314 (+ 1575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681369}
I20251029 08:52:28.374193 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 43c3074caee74069a8a7f1b87721d726 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]) successfully deleted
I20251029 08:52:28.374606 1179 tablet_replica.cc:333] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.374884 1179 raft_consensus.cc:2243] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:28.376894 921 raft_consensus.cc:1275] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Refusing update from remote peer 730d0f5b7abf43a293b7df13d01ce4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:28.378311 2451 leader_election.cc:290] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.378532 1179 raft_consensus.cc:2272] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.379037 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3955ff79faf42bb82f764aae79b85d9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.379256 1067 raft_consensus.cc:2468] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:28.379220 2707 consensus_queue.cc:1048] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:28.380204 861 leader_election.cc:304] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:28.380504 861 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:28.380870 2451 raft_consensus.cc:2804] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.380942 2451 raft_consensus.cc:493] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.381000 2451 raft_consensus.cc:3060] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.381336 902 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5eee4655b9445cb2c05c3a0307cee2 (DEFAULT_TABLE table=default.Table_980d17e6af4a4439b1f459070584dcec [id=595c8cb5dac04a1bb7f151832c8e702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.381599 1974 raft_consensus.cc:2272] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.382550 1179 ts_tablet_manager.cc:1916] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.386722 1974 ts_tablet_manager.cc:1916] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.393710 1179 ts_tablet_manager.cc:1929] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.397089 1177 tablet_bootstrap.cc:654] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.398343 1179 log.cc:1199] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:28.399911 1179 ts_tablet_manager.cc:1950] T a8c4d57a59684723bfd27c7dda302ef3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.401036 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 998) took 2739 ms. Trace:
I20251029 08:52:28.408324 1179 rpcz_store.cc:276] 1029 08:52:25.661772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.661814 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:28.374523 (+2712709us) ts_tablet_manager.cc:1151] Deleting tablet a8c4d57a59684723bfd27c7dda302ef3
1029 08:52:28.374538 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.386695 (+ 12157us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.393570 (+ 6875us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.400991 (+ 7421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2712567}
I20251029 08:52:28.408959 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a8c4d57a59684723bfd27c7dda302ef3 (table default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]) successfully deleted
I20251029 08:52:28.409436 1179 tablet_replica.cc:333] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.409696 1179 raft_consensus.cc:2243] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.409960 1179 raft_consensus.cc:2272] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.410876 1179 ts_tablet_manager.cc:1916] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.411442 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5eee4655b9445cb2c05c3a0307cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.414949 2707 raft_consensus.cc:493] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.415110 2707 raft_consensus.cc:515] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:28.415740 2707 leader_election.cc:290] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.416482 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d6d720fb6b54b1e8bb7b3956c48b3f9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:28.416923 998 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 6d6d720fb6b54b1e8bb7b3956c48b3f9
W20251029 08:52:28.417202 996 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.417318 996 leader_election.cc:304] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.417838 2707 raft_consensus.cc:2749] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.421473 1179 ts_tablet_manager.cc:1929] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.424216 2451 raft_consensus.cc:515] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.424898 2451 leader_election.cc:290] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.425088 1179 log.cc:1199] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aa4f97075b3941aaa7cda3f9413e867f
W20251029 08:52:28.426153 861 leader_election.cc:343] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:28.425570 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3955ff79faf42bb82f764aae79b85d9" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:28.432192 1067 raft_consensus.cc:3060] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:28.435277 861 consensus_peers.cc:597] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39fc9612689b46d498d6092bd552da4e. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.435318 1067 raft_consensus.cc:2468] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:28.435909 861 leader_election.cc:304] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.436317 2451 raft_consensus.cc:2804] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:28.436472 2451 raft_consensus.cc:697] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:28.436712 2451 consensus_queue.cc:237] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.437000 1179 ts_tablet_manager.cc:1950] T aa4f97075b3941aaa7cda3f9413e867f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.438007 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 999) took 2775 ms. Trace:
I20251029 08:52:28.438191 1179 rpcz_store.cc:276] 1029 08:52:25.662663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.662703 (+ 40us) service_pool.cc:225] Handling call
1029 08:52:28.409330 (+2746627us) ts_tablet_manager.cc:1151] Deleting tablet aa4f97075b3941aaa7cda3f9413e867f
1029 08:52:28.409346 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.416535 (+ 7189us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.421312 (+ 4777us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.437957 (+ 16645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2746466}
I20251029 08:52:28.440398 752 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb0edbb3d7b43728b9ade05665c6972 (DEFAULT_TABLE table=default.Table_1d1ac2a940e44574bb336cd7805b6cf1 [id=20c50f915b1a493b958e41763acbd27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.440879 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb0edbb3d7b43728b9ade05665c6972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.441210 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aa4f97075b3941aaa7cda3f9413e867f (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
I20251029 08:52:28.440063 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5eee4655b9445cb2c05c3a0307cee2 (DEFAULT_TABLE table=default.Table_980d17e6af4a4439b1f459070584dcec [id=595c8cb5dac04a1bb7f151832c8e702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.441676 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5eee4655b9445cb2c05c3a0307cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.442513 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13d40f6bbc974e0bb9e249282fe3a279" } modify_external_catalogs: true
I20251029 08:52:28.442736 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_13d40f6bbc974e0bb9e249282fe3a279" } modify_external_catalogs: true
I20251029 08:52:28.448092 1179 tablet_replica.cc:333] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.448794 1179 raft_consensus.cc:2243] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.449118 1179 raft_consensus.cc:2272] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.449838 1179 ts_tablet_manager.cc:1916] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:28.454650 861 consensus_peers.cc:597] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b07eb4befa41f3a1b5f2e9ebc2277c. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.459331 1177 tablet_bootstrap.cc:492] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:28.459528 1177 ts_tablet_manager.cc:1403] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.121s user 0.000s sys 0.010s
I20251029 08:52:28.460528 1177 raft_consensus.cc:359] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.460778 1177 raft_consensus.cc:385] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.460862 1177 raft_consensus.cc:740] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.461094 1177 consensus_queue.cc:260] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.461933 1177 ts_tablet_manager.cc:1434] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:28.462281 1177 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:28.465240 2707 raft_consensus.cc:493] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.465420 2707 raft_consensus.cc:515] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:28.466288 2707 leader_election.cc:290] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:28.467543 998 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:28.467779 2295 raft_consensus.cc:493] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.467901 2295 raft_consensus.cc:515] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.468075 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9429f6abb50c4f66b2e1b565f303962d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.468518 2295 leader_election.cc:290] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:28.469071 996 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.469198 996 leader_election.cc:304] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:28.470067 861 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:28.470582 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fdc381a0d949b0b93cef399daa3021" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.470800 1067 raft_consensus.cc:2468] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
I20251029 08:52:28.471246 861 leader_election.cc:304] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.471658 2707 raft_consensus.cc:2749] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.471837 2731 raft_consensus.cc:2804] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.471906 2731 raft_consensus.cc:493] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.471958 2731 raft_consensus.cc:3060] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.475248 2731 raft_consensus.cc:515] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.475904 2731 leader_election.cc:290] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:28.478113 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fdc381a0d949b0b93cef399daa3021" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:28.478317 1067 raft_consensus.cc:3060] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.482612 1067 raft_consensus.cc:2468] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
W20251029 08:52:28.483481 861 leader_election.cc:343] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:28.483764 861 leader_election.cc:304] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:28.484463 2554 raft_consensus.cc:2804] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:28.484596 2554 raft_consensus.cc:697] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:28.484853 2554 consensus_queue.cc:237] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.493227 1045 tablet_service.cc:1505] Processing CreateTablet for tablet e57e4313adce4e35843ccd9ee137e8f7 (DEFAULT_TABLE table=default.Table_d29aed5fe8f44c47a0f174f5d5fd9d5f [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.493784 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57e4313adce4e35843ccd9ee137e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.497628 752 tablet_service.cc:1505] Processing CreateTablet for tablet a711e95fbef84b12997d01c1883057d9 (DEFAULT_TABLE table=default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.498081 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a711e95fbef84b12997d01c1883057d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.499248 1047 tablet_service.cc:1505] Processing CreateTablet for tablet a711e95fbef84b12997d01c1883057d9 (DEFAULT_TABLE table=default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.499472 1032 tablet_service.cc:1505] Processing CreateTablet for tablet dab9987dba674b198130e07a79eb5028 (DEFAULT_TABLE table=default.Table_6917eb552d404f19b2e477c77b0fcf51 [id=a9c95c1708244b1e8e4d62fbf1519f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.499689 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a711e95fbef84b12997d01c1883057d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.500311 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab9987dba674b198130e07a79eb5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.500862 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e14d1d3421a542ec960d21453bb713ec
I20251029 08:52:28.503898 593 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_afa011ccbb98416495ed91cfb02d50fb [id=32fdba980db34781a7f359080cb1e56c]). Expected version 3 got 0
I20251029 08:52:28.504325 1044 tablet_service.cc:1505] Processing CreateTablet for tablet d51192ae278d46d4b169e2ffd76e9d7c (DEFAULT_TABLE table=default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.504577 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d1d3421a542ec960d21453bb713ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:28.504765 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51192ae278d46d4b169e2ffd76e9d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.506214 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_306a00708d9448639fca68a831b3bef3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:28.506747 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_306a00708d9448639fca68a831b3bef3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:28.508074 590 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_afa011ccbb98416495ed91cfb02d50fb [id=32fdba980db34781a7f359080cb1e56c]). Expected version 3 got 0
I20251029 08:52:28.508298 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 580bc00ba27f44e3961f0cd91b1c83c5 (DEFAULT_TABLE table=default.Table_0a69ab65d26044739b602d6040ee95a3 [id=a9e19c9b7bc1463f85367a4afde02e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.508689 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580bc00ba27f44e3961f0cd91b1c83c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.509033 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_72b06540aa1c4b51826638ef3543665e" } new_table_name: "default.Table_f861cefe281a475fbd4ef5e56a0f7b8f" modify_external_catalogs: true
I20251029 08:52:28.499274 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1f454b26ae4ec3a946245e0a565ece (DEFAULT_TABLE table=default.Table_0d1d1cca556e4e6c91159f6cbbf6a2ec [id=3242398a99134b2abd20ebee1cd8f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.510782 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.511067 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.511125 747 tablet_service.cc:1505] Processing CreateTablet for tablet d51192ae278d46d4b169e2ffd76e9d7c (DEFAULT_TABLE table=default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.511511 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51192ae278d46d4b169e2ffd76e9d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.516382 758 tablet_service.cc:1505] Processing CreateTablet for tablet dab9987dba674b198130e07a79eb5028 (DEFAULT_TABLE table=default.Table_6917eb552d404f19b2e477c77b0fcf51 [id=a9c95c1708244b1e8e4d62fbf1519f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.516786 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab9987dba674b198130e07a79eb5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.520157 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d1d3421a542ec960d21453bb713ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.520355 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:28.520805 757 tablet_service.cc:1505] Processing CreateTablet for tablet 580bc00ba27f44e3961f0cd91b1c83c5 (DEFAULT_TABLE table=default.Table_0a69ab65d26044739b602d6040ee95a3 [id=a9e19c9b7bc1463f85367a4afde02e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.521181 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580bc00ba27f44e3961f0cd91b1c83c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.523597 889 tablet_service.cc:1505] Processing CreateTablet for tablet 580bc00ba27f44e3961f0cd91b1c83c5 (DEFAULT_TABLE table=default.Table_0a69ab65d26044739b602d6040ee95a3 [id=a9e19c9b7bc1463f85367a4afde02e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.523859 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e14d1d3421a542ec960d21453bb713ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:28.524318 904 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb0edbb3d7b43728b9ade05665c6972 (DEFAULT_TABLE table=default.Table_1d1ac2a940e44574bb336cd7805b6cf1 [id=20c50f915b1a493b958e41763acbd27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.524740 902 tablet_service.cc:1505] Processing CreateTablet for tablet e57e4313adce4e35843ccd9ee137e8f7 (DEFAULT_TABLE table=default.Table_d29aed5fe8f44c47a0f174f5d5fd9d5f [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.525336 886 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1f454b26ae4ec3a946245e0a565ece (DEFAULT_TABLE table=default.Table_0d1d1cca556e4e6c91159f6cbbf6a2ec [id=3242398a99134b2abd20ebee1cd8f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.527668 898 tablet_service.cc:1505] Processing CreateTablet for tablet a711e95fbef84b12997d01c1883057d9 (DEFAULT_TABLE table=default.Table_4e34d668a8db45a9b820969bb5939129 [id=334d077ad2634b9490979d41a64e7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.538411 1178 tablet_bootstrap.cc:492] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:28.538764 1178 ts_tablet_manager.cc:1403] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.400s user 0.030s sys 0.058s
I20251029 08:52:28.541620 2756 tablet.cc:1722] T a4c74b052ac94eae9e226285504c6b40 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:28.541632 1178 raft_consensus.cc:359] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.542239 1178 raft_consensus.cc:385] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.542595 1178 raft_consensus.cc:740] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.543810 1178 consensus_queue.cc:260] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.546922 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1f454b26ae4ec3a946245e0a565ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.558568 1179 ts_tablet_manager.cc:1929] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.558787 1179 log.cc:1199] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:28.559240 1179 ts_tablet_manager.cc:1950] T 627bc8ff99394fa2abcca15d62e5ea10 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.560060 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb0edbb3d7b43728b9ade05665c6972 (DEFAULT_TABLE table=default.Table_1d1ac2a940e44574bb336cd7805b6cf1 [id=20c50f915b1a493b958e41763acbd27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.560238 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1000) took 2896 ms. Trace:
I20251029 08:52:28.560346 1179 rpcz_store.cc:276] 1029 08:52:25.663322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.663365 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:28.447988 (+2784623us) ts_tablet_manager.cc:1151] Deleting tablet 627bc8ff99394fa2abcca15d62e5ea10
1029 08:52:28.447998 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.461092 (+ 13094us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.541932 (+ 80840us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.560191 (+ 18259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2784481}
I20251029 08:52:28.560545 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb0edbb3d7b43728b9ade05665c6972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.560721 1179 tablet_replica.cc:333] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.560921 1179 raft_consensus.cc:2243] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.561116 1179 raft_consensus.cc:2272] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.562604 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 627bc8ff99394fa2abcca15d62e5ea10 (table default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]) successfully deleted
I20251029 08:52:28.565207 1179 ts_tablet_manager.cc:1916] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.565579 1178 ts_tablet_manager.cc:1434] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20251029 08:52:28.566938 2755 tablet.cc:1722] T a4c74b052ac94eae9e226285504c6b40 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:28.567917 1178 tablet_bootstrap.cc:492] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:28.572984 1183 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:28.573159 1183 ts_tablet_manager.cc:1403] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.240s user 0.004s sys 0.002s
I20251029 08:52:28.573992 1183 raft_consensus.cc:359] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.574199 1183 raft_consensus.cc:385] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.574288 1183 raft_consensus.cc:740] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.574549 1183 consensus_queue.cc:260] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.575377 1183 ts_tablet_manager.cc:1434] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:28.578171 1183 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:28.581794 747 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad6f423025a463e944a7f34ac222286 (DEFAULT_TABLE table=default.Table_3773b920e9b340ad8796f4a171e769a2 [id=a8ea586c8a50466d9956b3c6eb751e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.582284 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad6f423025a463e944a7f34ac222286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.584630 764 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1f454b26ae4ec3a946245e0a565ece (DEFAULT_TABLE table=default.Table_0d1d1cca556e4e6c91159f6cbbf6a2ec [id=3242398a99134b2abd20ebee1cd8f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.585538 896 tablet_service.cc:1505] Processing CreateTablet for tablet dab9987dba674b198130e07a79eb5028 (DEFAULT_TABLE table=default.Table_6917eb552d404f19b2e477c77b0fcf51 [id=a9c95c1708244b1e8e4d62fbf1519f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.586452 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad6f423025a463e944a7f34ac222286 (DEFAULT_TABLE table=default.Table_3773b920e9b340ad8796f4a171e769a2 [id=a8ea586c8a50466d9956b3c6eb751e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.586962 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad6f423025a463e944a7f34ac222286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.596581 1179 ts_tablet_manager.cc:1929] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.598166 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1f454b26ae4ec3a946245e0a565ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.600045 1179 log.cc:1199] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4aa0ff14e19b4ad7a8b33796481f0f3f
I20251029 08:52:28.600780 1178 tablet_bootstrap.cc:654] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.601255 1179 ts_tablet_manager.cc:1950] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.598094 1051 tablet_service.cc:1505] Processing CreateTablet for tablet f744556dccf348fd8b92071f224730aa (DEFAULT_TABLE table=default.Table_4cf1783778c64509b2a98c717f0477a2 [id=db574ec9678e4ee59e36b8e59ee7b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.601938 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f744556dccf348fd8b92071f224730aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.602067 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1001) took 2938 ms. Trace:
I20251029 08:52:28.605055 1179 rpcz_store.cc:276] 1029 08:52:25.663880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.663930 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:28.560646 (+2896716us) ts_tablet_manager.cc:1151] Deleting tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
1029 08:52:28.560655 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.586504 (+ 25849us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.596463 (+ 9959us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.602022 (+ 5559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896556}
I20251029 08:52:28.605787 1179 tablet_replica.cc:333] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.605986 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (table default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]) successfully deleted
I20251029 08:52:28.606304 1179 raft_consensus.cc:2243] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.606518 1179 raft_consensus.cc:2272] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.597949 1183 tablet_bootstrap.cc:654] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.613694 1179 ts_tablet_manager.cc:1916] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.618973 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 3915fd705caa420eb64879ea513b8cea (DEFAULT_TABLE table=default.Table_84ecf561646e4d788d12eccac9cb8646 [id=71556ffb46c24b0db3457ba8f860abb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.619764 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915fd705caa420eb64879ea513b8cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.634167 757 tablet_service.cc:1505] Processing CreateTablet for tablet f744556dccf348fd8b92071f224730aa (DEFAULT_TABLE table=default.Table_4cf1783778c64509b2a98c717f0477a2 [id=db574ec9678e4ee59e36b8e59ee7b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.635596 895 tablet_service.cc:1505] Processing CreateTablet for tablet d51192ae278d46d4b169e2ffd76e9d7c (DEFAULT_TABLE table=default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.637669 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f744556dccf348fd8b92071f224730aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.648191 758 tablet_service.cc:1505] Processing CreateTablet for tablet 3915fd705caa420eb64879ea513b8cea (DEFAULT_TABLE table=default.Table_84ecf561646e4d788d12eccac9cb8646 [id=71556ffb46c24b0db3457ba8f860abb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.648653 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915fd705caa420eb64879ea513b8cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.654784 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb0edbb3d7b43728b9ade05665c6972. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:28.655225 861 consensus_peers.cc:597] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627bc8ff99394fa2abcca15d62e5ea10. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.659654 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57e4313adce4e35843ccd9ee137e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.662609 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1f454b26ae4ec3a946245e0a565ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.664137 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab9987dba674b198130e07a79eb5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.667589 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a711e95fbef84b12997d01c1883057d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.671283 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580bc00ba27f44e3961f0cd91b1c83c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.680511 1046 tablet_service.cc:1505] Processing CreateTablet for tablet adbff30ec60c4e559f95ddb21b60e232 (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.680994 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbff30ec60c4e559f95ddb21b60e232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.685542 894 tablet_service.cc:1505] Processing CreateTablet for tablet f744556dccf348fd8b92071f224730aa (DEFAULT_TABLE table=default.Table_4cf1783778c64509b2a98c717f0477a2 [id=db574ec9678e4ee59e36b8e59ee7b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.686113 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3915fd705caa420eb64879ea513b8cea (DEFAULT_TABLE table=default.Table_84ecf561646e4d788d12eccac9cb8646 [id=71556ffb46c24b0db3457ba8f860abb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.686147 749 tablet_service.cc:1505] Processing CreateTablet for tablet e57e4313adce4e35843ccd9ee137e8f7 (DEFAULT_TABLE table=default.Table_d29aed5fe8f44c47a0f174f5d5fd9d5f [id=d20448292e8c40109ee4c584de13ea95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.686553 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57e4313adce4e35843ccd9ee137e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.686578 899 tablet_service.cc:1505] Processing CreateTablet for tablet adbff30ec60c4e559f95ddb21b60e232 (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.687021 897 tablet_service.cc:1505] Processing CreateTablet for tablet c5d9527d294c432db2ccf60af84731b0 (DEFAULT_TABLE table=default.Table_b6d28fc5fdc14b10b3436923d1b86bc1 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.687460 891 tablet_service.cc:1505] Processing CreateTablet for tablet 8bc9ac40dd5b49578945e3e68e235c04 (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.692080 905 tablet_service.cc:1505] Processing CreateTablet for tablet 26cc2d5ccb7e49ab92c8b1da09152b27 (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.693991 893 tablet_service.cc:1505] Processing CreateTablet for tablet 101aff0682d747dca3846bf95bce0a5d (DEFAULT_TABLE table=default.Table_17251c1e0e6447b0bb5b5cf8c78e42ed [id=9ad20e5674744960b1c9f1f965e52bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.694523 1183 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:28.694686 1183 ts_tablet_manager.cc:1403] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.002s
I20251029 08:52:28.695588 1183 raft_consensus.cc:359] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.695786 1183 raft_consensus.cc:385] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.695861 1183 raft_consensus.cc:740] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.696138 1183 consensus_queue.cc:260] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:28.697045 861 consensus_peers.cc:597] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa411391c63146e7acb8b4c6edbe12a0. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:28.699416 861 consensus_peers.cc:597] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 652903cbbdb042d780e7d71fc2c0ab4d. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:28.702307 861 consensus_peers.cc:597] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.703190 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c5d9527d294c432db2ccf60af84731b0 (DEFAULT_TABLE table=default.Table_b6d28fc5fdc14b10b3436923d1b86bc1 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.703266 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 8bc9ac40dd5b49578945e3e68e235c04 (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.703619 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d9527d294c432db2ccf60af84731b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.704077 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc9ac40dd5b49578945e3e68e235c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.705835 750 tablet_service.cc:1505] Processing CreateTablet for tablet 8bc9ac40dd5b49578945e3e68e235c04 (DEFAULT_TABLE table=default.Table_20e164b8395a4054a8c704d344a20a57 [id=66eee56ac86141938732092093b9e0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.706218 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc9ac40dd5b49578945e3e68e235c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.707552 747 tablet_service.cc:1505] Processing CreateTablet for tablet 26cc2d5ccb7e49ab92c8b1da09152b27 (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.707679 753 tablet_service.cc:1505] Processing CreateTablet for tablet c5d9527d294c432db2ccf60af84731b0 (DEFAULT_TABLE table=default.Table_b6d28fc5fdc14b10b3436923d1b86bc1 [id=572a15f289894adb908547d8f011765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.707924 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cc2d5ccb7e49ab92c8b1da09152b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.708130 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d9527d294c432db2ccf60af84731b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.712451 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 101aff0682d747dca3846bf95bce0a5d (DEFAULT_TABLE table=default.Table_17251c1e0e6447b0bb5b5cf8c78e42ed [id=9ad20e5674744960b1c9f1f965e52bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.712934 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101aff0682d747dca3846bf95bce0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.721221 745 tablet_service.cc:1505] Processing CreateTablet for tablet 101aff0682d747dca3846bf95bce0a5d (DEFAULT_TABLE table=default.Table_17251c1e0e6447b0bb5b5cf8c78e42ed [id=9ad20e5674744960b1c9f1f965e52bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.722688 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101aff0682d747dca3846bf95bce0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.725013 1179 ts_tablet_manager.cc:1929] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.725208 1179 log.cc:1199] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:28.725657 1179 ts_tablet_manager.cc:1950] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.726464 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1002) took 3061 ms. Trace:
I20251029 08:52:28.726629 1179 rpcz_store.cc:276] 1029 08:52:25.664514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.664558 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:28.605715 (+2941157us) ts_tablet_manager.cc:1151] Deleting tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
1029 08:52:28.605726 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.617608 (+ 11882us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.636065 (+ 18457us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.726413 (+ 90348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2941024}
I20251029 08:52:28.727047 1179 tablet_replica.cc:333] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.727291 1179 raft_consensus.cc:2243] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.727521 1179 raft_consensus.cc:2272] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.732594 1179 ts_tablet_manager.cc:1916] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.733065 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 26cc2d5ccb7e49ab92c8b1da09152b27 (DEFAULT_TABLE table=default.Table_8a015c8b3b8b49a7a49ae2c3694ab71b [id=02cd47f877e948089667000033684113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.733570 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cc2d5ccb7e49ab92c8b1da09152b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.744756 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=875 overwritten=0 applied=874 ignored=0} inserts{seen=816 ignored=0} mutations{seen=1003 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251029 08:52:28.745808 2395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: Bootstrap complete.
I20251029 08:52:28.752107 2756 tablet.cc:1722] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:28.767048 1179 ts_tablet_manager.cc:1929] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.767457 1179 log.cc:1199] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/ba201a54466240129d82a096f9316242
I20251029 08:52:28.768482 1179 ts_tablet_manager.cc:1950] T ba201a54466240129d82a096f9316242 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.768883 758 tablet_service.cc:1505] Processing CreateTablet for tablet adbff30ec60c4e559f95ddb21b60e232 (DEFAULT_TABLE table=default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.770463 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1003) took 3105 ms. Trace:
I20251029 08:52:28.770960 1179 rpcz_store.cc:276] 1029 08:52:25.665191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.665236 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:28.726982 (+3061746us) ts_tablet_manager.cc:1151] Deleting tablet ba201a54466240129d82a096f9316242
1029 08:52:28.726991 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.755447 (+ 28456us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.766857 (+ 11410us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.770395 (+ 3538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3061620}
I20251029 08:52:28.771133 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbff30ec60c4e559f95ddb21b60e232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.772807 1179 tablet_replica.cc:333] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.773240 1179 raft_consensus.cc:2243] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:28.774348 1183 ts_tablet_manager.cc:1434] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.071s user 0.003s sys 0.000s
I20251029 08:52:28.776212 1179 raft_consensus.cc:2272] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.777194 1179 ts_tablet_manager.cc:1916] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:28.788604 753 tablet_service.cc:1505] Processing CreateTablet for tablet 21d409c2e24947beb99351da46a5aef9 (DEFAULT_TABLE table=default.Table_7f6426ed22804c409dd431e21ae1d0ca [id=c964cd9f9fc544d4999ff2c5800b5112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.789038 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d409c2e24947beb99351da46a5aef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.792662 2395 raft_consensus.cc:359] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:28.795023 2761 tablet.cc:1722] T a4c74b052ac94eae9e226285504c6b40 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 08:52:28.795392 902 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad6f423025a463e944a7f34ac222286 (DEFAULT_TABLE table=default.Table_3773b920e9b340ad8796f4a171e769a2 [id=a8ea586c8a50466d9956b3c6eb751e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.800436 750 tablet_service.cc:1505] Processing CreateTablet for tablet 81236599ea1149bca1d1ef1a84e60c1a (DEFAULT_TABLE table=default.Table_4ccf023f7a3f4275bfb29b5983db651e [id=bdc529dd4e3a4326b3c89f93a9835087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.800966 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81236599ea1149bca1d1ef1a84e60c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.805876 2755 tablet.cc:1722] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:28.809626 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 81236599ea1149bca1d1ef1a84e60c1a (DEFAULT_TABLE table=default.Table_4ccf023f7a3f4275bfb29b5983db651e [id=bdc529dd4e3a4326b3c89f93a9835087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.810067 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81236599ea1149bca1d1ef1a84e60c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.823477 2395 raft_consensus.cc:740] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f718bd0f00004f20a89d028a2fa8e6b9, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.825997 2395 consensus_queue.cc:260] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 874, Last appended: 3.875, Last appended by leader: 875, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:28.839777 2395 sys_catalog.cc:565] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 08:52:28.839699 2763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:28.842615 886 tablet_service.cc:1505] Processing CreateTablet for tablet 81236599ea1149bca1d1ef1a84e60c1a (DEFAULT_TABLE table=default.Table_4ccf023f7a3f4275bfb29b5983db651e [id=bdc529dd4e3a4326b3c89f93a9835087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.843468 896 tablet_service.cc:1505] Processing CreateTablet for tablet d664e6b3d8474a03b544efc333d1bab6 (DEFAULT_TABLE table=default.Table_fc24bd6f44c0439f9a83693d672608a2 [id=b7bebc5cb2fa488fb2db8637be13c5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.844182 1179 ts_tablet_manager.cc:1929] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:28.844336 1179 log.cc:1199] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:28.844199 904 tablet_service.cc:1505] Processing CreateTablet for tablet 21d409c2e24947beb99351da46a5aef9 (DEFAULT_TABLE table=default.Table_7f6426ed22804c409dd431e21ae1d0ca [id=c964cd9f9fc544d4999ff2c5800b5112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.844755 1179 ts_tablet_manager.cc:1950] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:28.845630 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1004) took 3179 ms. Trace:
I20251029 08:52:28.845743 1179 rpcz_store.cc:276] 1029 08:52:25.665818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.665875 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:28.772003 (+3106128us) ts_tablet_manager.cc:1151] Deleting tablet 5f5ebd2bf6614980ac0c18f5d9974fc0
1029 08:52:28.772048 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.810263 (+ 38215us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.844036 (+ 33773us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:28.845582 (+ 1546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3105989}
I20251029 08:52:28.846204 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1010) took 3096 ms. Trace:
I20251029 08:52:28.846319 1179 rpcz_store.cc:276] 1029 08:52:25.749755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.749821 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:28.846039 (+3096218us) ts_tablet_manager.cc:1151] Deleting tablet 0c2072c16fce43278f1e32b9672187e7
1029 08:52:28.846051 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.846158 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096006}
I20251029 08:52:28.846717 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1011) took 3088 ms. Trace:
I20251029 08:52:28.846822 1179 rpcz_store.cc:276] 1029 08:52:25.758358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.758411 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:28.846585 (+3088174us) ts_tablet_manager.cc:1151] Deleting tablet 41622daeb1e0419fa88cc26c0405c2db
1029 08:52:28.846595 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.846672 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088019}
I20251029 08:52:28.847313 1183 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:28.848028 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1012) took 3089 ms. Trace:
I20251029 08:52:28.848137 1179 rpcz_store.cc:276] 1029 08:52:25.758511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.758599 (+ 88us) service_pool.cc:225] Handling call
1029 08:52:28.847882 (+3089283us) ts_tablet_manager.cc:1151] Deleting tablet 5498826a9afd4fd5a6b88b0a465fff40
1029 08:52:28.847894 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.847975 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088286}
I20251029 08:52:28.848516 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1015) took 3035 ms. Trace:
I20251029 08:52:28.848628 1179 rpcz_store.cc:276] 1029 08:52:25.812759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.812956 (+ 197us) service_pool.cc:225] Handling call
1029 08:52:28.848379 (+3035423us) ts_tablet_manager.cc:1151] Deleting tablet 52b697b962f849eb88741c2e2ba4643d
1029 08:52:28.848389 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.848477 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035209}
I20251029 08:52:28.848980 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1016) took 3036 ms. Trace:
I20251029 08:52:28.849085 1179 rpcz_store.cc:276] 1029 08:52:25.812894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.813102 (+ 208us) service_pool.cc:225] Handling call
1029 08:52:28.848853 (+3035751us) ts_tablet_manager.cc:1151] Deleting tablet 58b02839973e4f018a1a6dfccf009b25
1029 08:52:28.848863 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.848937 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035582}
W20251029 08:52:28.849339 861 consensus_peers.cc:597] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7be4bcf9e3324bd884e946fae20648fd. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.849490 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1013) took 3068 ms. Trace:
I20251029 08:52:28.849609 1179 rpcz_store.cc:276] 1029 08:52:25.780812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.838029 (+ 57217us) service_pool.cc:225] Handling call
1029 08:52:28.849360 (+3011331us) ts_tablet_manager.cc:1151] Deleting tablet 2fb9ab76dcfe486e89bbe9391cab56f9
1029 08:52:28.849371 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.849441 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969449}
I20251029 08:52:28.841786 2763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:28.850011 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1020) took 2906 ms. Trace:
I20251029 08:52:28.850165 1179 rpcz_store.cc:276] 1029 08:52:25.943714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.954421 (+ 10707us) service_pool.cc:225] Handling call
1029 08:52:28.849892 (+2895471us) ts_tablet_manager.cc:1151] Deleting tablet 0c14c63f74f74a18902a17a4d210a90d
1029 08:52:28.849900 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.849974 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2895196}
I20251029 08:52:28.850513 1179 tablet_replica.cc:333] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:28.850775 1179 raft_consensus.cc:2243] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:28.850989 1179 raft_consensus.cc:2272] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:28.851764 1179 ts_tablet_manager.cc:1916] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:28.853334 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 69145.6us
I20251029 08:52:28.853619 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:28.854075 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet ba201a54466240129d82a096f9316242 (table default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]) successfully deleted
I20251029 08:52:28.855840 1183 tablet_bootstrap.cc:654] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:28.863216 2742 raft_consensus.cc:493] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.863360 2742 raft_consensus.cc:515] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.864009 2742 leader_election.cc:290] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.864897 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3a2fab467348099a4a1fec74eb5e65" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.864970 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3a2fab467348099a4a1fec74eb5e65" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:28.865896 720 leader_election.cc:343] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:28.866878 718 leader_election.cc:343] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.867056 718 leader_election.cc:304] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:28.867383 2742 raft_consensus.cc:2749] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:28.875303 1178 tablet_bootstrap.cc:492] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:28.875509 1178 ts_tablet_manager.cc:1403] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.308s user 0.066s sys 0.071s
I20251029 08:52:28.876323 1178 raft_consensus.cc:359] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.876818 1178 raft_consensus.cc:385] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:28.878154 1178 raft_consensus.cc:740] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:28.880688 764 tablet_service.cc:1505] Processing CreateTablet for tablet d664e6b3d8474a03b544efc333d1bab6 (DEFAULT_TABLE table=default.Table_fc24bd6f44c0439f9a83693d672608a2 [id=b7bebc5cb2fa488fb2db8637be13c5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.881206 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e6b3d8474a03b544efc333d1bab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.881431 2742 raft_consensus.cc:493] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:28.881554 2742 raft_consensus.cc:515] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.882125 2742 leader_election.cc:290] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.882572 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:28.882922 1178 consensus_queue.cc:260] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:28.885875 861 consensus_peers.cc:597] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa4f97075b3941aaa7cda3f9413e867f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.885946 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:28.886142 1065 raft_consensus.cc:2468] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
I20251029 08:52:28.886461 720 leader_election.cc:304] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:28.886727 2742 raft_consensus.cc:2804] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:28.886786 2742 raft_consensus.cc:493] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:28.886837 2742 raft_consensus.cc:3060] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.887756 1047 tablet_service.cc:1505] Processing CreateTablet for tablet d664e6b3d8474a03b544efc333d1bab6 (DEFAULT_TABLE table=default.Table_fc24bd6f44c0439f9a83693d672608a2 [id=b7bebc5cb2fa488fb2db8637be13c5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.888192 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e6b3d8474a03b544efc333d1bab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:28.889511 2742 raft_consensus.cc:515] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:28.890017 2742 leader_election.cc:290] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:28.891989 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:28.892203 1065 raft_consensus.cc:3060] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:28.895190 1178 ts_tablet_manager.cc:1434] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
W20251029 08:52:28.895963 718 leader_election.cc:343] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.898207 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce4a3503830c481797133028f5f123d9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:28.900429 718 leader_election.cc:343] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:28.906636 1178 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:28.915874 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 60493us
I20251029 08:52:28.916934 901 tablet_service.cc:1505] Processing CreateTablet for tablet dd393400e8b2473482ab78008863c48b (DEFAULT_TABLE table=default.Table_073a0e8a44f04947b524dac0cdb01fb1 [id=754bb7bba118445b9a26e893b09071f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:28.924263 2295 consensus_queue.cc:1048] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:28.937381 861 consensus_peers.cc:597] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9bbf6401423411089c9070f3b03f1a7. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.949513 1178 tablet_bootstrap.cc:654] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:28.973724 861 consensus_peers.cc:597] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e213713757949caa3f504f70026b7b5. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:28.996976 2451 consensus_queue.cc:1048] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:29.001852 2764 tablet.cc:1722] T 6a7091ff6bed4e3a8447d9de8ea9fb48 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:29.003312 1176 ts_tablet_manager.cc:1929] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.003650 1176 log.cc:1199] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/59ad04f57c29407b9d2420520c095a28
I20251029 08:52:28.941145 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (table default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]) successfully deleted
W20251029 08:52:29.005571 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 67222.7us
W20251029 08:52:29.006079 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0c2072c16fce43278f1e32b9672187e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c2072c16fce43278f1e32b9672187e7 already in progress: creating tablet
W20251029 08:52:29.006374 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 41622daeb1e0419fa88cc26c0405c2db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41622daeb1e0419fa88cc26c0405c2db already in progress: creating tablet
W20251029 08:52:29.007076 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 52b697b962f849eb88741c2e2ba4643d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52b697b962f849eb88741c2e2ba4643d already in progress: creating tablet
W20251029 08:52:29.007339 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 58b02839973e4f018a1a6dfccf009b25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58b02839973e4f018a1a6dfccf009b25 already in progress: creating tablet
W20251029 08:52:29.007761 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb9ab76dcfe486e89bbe9391cab56f9 already in progress: creating tablet
W20251029 08:52:29.008170 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0c14c63f74f74a18902a17a4d210a90d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c14c63f74f74a18902a17a4d210a90d already in progress: creating tablet
I20251029 08:52:29.008386 2485 raft_consensus.cc:493] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.008549 2485 raft_consensus.cc:515] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.009228 2485 leader_election.cc:290] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.009994 749 tablet_service.cc:1505] Processing CreateTablet for tablet dd393400e8b2473482ab78008863c48b (DEFAULT_TABLE table=default.Table_073a0e8a44f04947b524dac0cdb01fb1 [id=754bb7bba118445b9a26e893b09071f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.010419 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd393400e8b2473482ab78008863c48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.011073 1178 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.014653 1178 ts_tablet_manager.cc:1403] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.110s user 0.024s sys 0.011s
W20251029 08:52:28.988970 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07bf5d693be240f18bacccc81a6ea5d4 consensus state has no leader
W20251029 08:52:28.994124 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57c0ad83e3f54150850b6617a1781382 consensus state has no leader
I20251029 08:52:28.994287 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d1d2df9e1e744ed1a6cf549ab57ed266
W20251029 08:52:28.941287 582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 47167.8us
I20251029 08:52:29.036636 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 21d409c2e24947beb99351da46a5aef9 (DEFAULT_TABLE table=default.Table_7f6426ed22804c409dd431e21ae1d0ca [id=c964cd9f9fc544d4999ff2c5800b5112]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.036749 898 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.037194 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c24c0d32f74435bd9d28f2415e2c47" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:29.037776 996 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.045166 998 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e3c24c0d32f74435bd9d28f2415e2c47
I20251029 08:52:29.045295 998 leader_election.cc:304] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.045532 2485 raft_consensus.cc:2749] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.048138 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d409c2e24947beb99351da46a5aef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.049094 1179 ts_tablet_manager.cc:1929] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.049345 1179 log.cc:1199] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/24834985a71349b9865840e9b5537d72
I20251029 08:52:29.049415 1183 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.049616 1183 ts_tablet_manager.cc:1403] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.275s user 0.010s sys 0.003s
I20251029 08:52:29.049963 1179 ts_tablet_manager.cc:1950] T 24834985a71349b9865840e9b5537d72 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:29.050473 1183 raft_consensus.cc:359] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.050673 1183 raft_consensus.cc:385] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.050733 1183 raft_consensus.cc:740] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.050624 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1029) took 2917 ms. Trace:
I20251029 08:52:29.050962 1183 consensus_queue.cc:260] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.050956 1179 rpcz_store.cc:276] 1029 08:52:26.132912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.134280 (+ 1368us) service_pool.cc:225] Handling call
1029 08:52:28.850440 (+2716160us) ts_tablet_manager.cc:1151] Deleting tablet 24834985a71349b9865840e9b5537d72
1029 08:52:28.850448 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.908532 (+ 58084us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.048957 (+140425us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.050580 (+ 1623us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2715626}
I20251029 08:52:29.051719 1183 ts_tablet_manager.cc:1434] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:29.052001 1183 tablet_bootstrap.cc:492] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.052214 1179 tablet_replica.cc:333] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:29.052573 1179 raft_consensus.cc:2243] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:29.053171 1179 pending_rounds.cc:70] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:29.053318 1179 pending_rounds.cc:77] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037665305296896 op_type: NO_OP noop_request { }
I20251029 08:52:29.053570 1179 raft_consensus.cc:2889] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:29.053804 1179 raft_consensus.cc:2272] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.055032 1179 ts_tablet_manager.cc:1916] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.055812 1065 raft_consensus.cc:2468] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:29.056308 720 leader_election.cc:304] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.056777 2742 raft_consensus.cc:2804] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.058718 1176 ts_tablet_manager.cc:1950] T 59ad04f57c29407b9d2420520c095a28 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.059589 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1049) took 2817 ms. Trace:
I20251029 08:52:29.059697 1176 rpcz_store.cc:276] 1029 08:52:26.241807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.241942 (+ 135us) service_pool.cc:225] Handling call
1029 08:52:27.983119 (+1741177us) ts_tablet_manager.cc:1151] Deleting tablet 59ad04f57c29407b9d2420520c095a28
1029 08:52:27.983131 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:28.348907 (+365776us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.003187 (+654280us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.059565 (+ 56378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741033}
I20251029 08:52:29.067695 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4d8742cba3e14ea3b056d42485790edc" } new_table_name: "default.Table_5460cd338afa438ca019608e172f4313" modify_external_catalogs: true
I20251029 08:52:29.070919 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a83c7d283315417781f403f77ef76951" } new_table_name: "default.Table_534dc559b38642b2939a1ff2e6a8b690" modify_external_catalogs: true
I20251029 08:52:29.072256 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_06720bad084b461996fdf83b38285cd3" } modify_external_catalogs: true
I20251029 08:52:29.072392 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_06720bad084b461996fdf83b38285cd3" } modify_external_catalogs: true
I20251029 08:52:29.073031 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f21470e046804cd0b003d632d05de8b9" } modify_external_catalogs: true
I20251029 08:52:29.073160 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_f21470e046804cd0b003d632d05de8b9" } modify_external_catalogs: true
W20251029 08:52:29.074913 861 consensus_peers.cc:597] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24834985a71349b9865840e9b5537d72. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.075212 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e3c95dd5d9224e2d8bddacd4dc0be319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.075317 861 consensus_peers.cc:597] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba201a54466240129d82a096f9316242. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:29.075711 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3c95dd5d9224e2d8bddacd4dc0be319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.076254 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cc2d5ccb7e49ab92c8b1da09152b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.076473 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad6f423025a463e944a7f34ac222286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.077850 1177 tablet_bootstrap.cc:654] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.078540 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d9527d294c432db2ccf60af84731b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.078722 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51192ae278d46d4b169e2ffd76e9d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.079188 593 master_service.cc:954] ReplaceTablet: received request to replace tablet c9b611ae08a247d3ba02eb177c6b1b12 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.080238 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_03738991ceac4afc818190d455239870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.080662 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03738991ceac4afc818190d455239870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.083452 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a4c74b052ac94eae9e226285504c6b40 (table default.Table_afa011ccbb98416495ed91cfb02d50fb [id=32fdba980db34781a7f359080cb1e56c]). Expected version 3 got 0
I20251029 08:52:29.084178 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5e9da1c94f3e437d9b2e61d9a51a18a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.084602 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e9da1c94f3e437d9b2e61d9a51a18a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.090521 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e321154841c14ccb8ef00074b1573785" } modify_external_catalogs: true
I20251029 08:52:29.090665 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e321154841c14ccb8ef00074b1573785" } modify_external_catalogs: true
I20251029 08:52:29.091816 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b9370f9fc0ee4f829cbd096420cf8c53" } new_table_name: "default.Table_91a64fd82f164fd8aae36227545d0014" modify_external_catalogs: true
I20251029 08:52:29.092592 1183 tablet_bootstrap.cc:654] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:29.099979 861 consensus_peers.cc:597] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8c4d793f10147fba5e4f44474575bf0. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.100381 2742 raft_consensus.cc:697] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:29.100721 2742 consensus_queue.cc:237] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.100912 1050 tablet_service.cc:1505] Processing CreateTablet for tablet dd393400e8b2473482ab78008863c48b (DEFAULT_TABLE table=default.Table_073a0e8a44f04947b524dac0cdb01fb1 [id=754bb7bba118445b9a26e893b09071f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.101267 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd393400e8b2473482ab78008863c48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.101696 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101aff0682d747dca3846bf95bce0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.105747 1179 ts_tablet_manager.cc:1929] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:29.109134 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd393400e8b2473482ab78008863c48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.109292 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d409c2e24947beb99351da46a5aef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.110961 1176 tablet_replica.cc:333] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.111176 1176 raft_consensus.cc:2243] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.111348 1176 raft_consensus.cc:2272] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.112455 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 59ad04f57c29407b9d2420520c095a28 (table default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]) successfully deleted
I20251029 08:52:29.115984 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 24834985a71349b9865840e9b5537d72 (table default.Table_0b00cd7c457d4204965de864c88be5e6 [id=0158571f3f9246aeafbfd8b8b1548482]) successfully deleted
I20251029 08:52:29.117384 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f744556dccf348fd8b92071f224730aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.120117 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81236599ea1149bca1d1ef1a84e60c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.122148 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbff30ec60c4e559f95ddb21b60e232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.123788 2742 raft_consensus.cc:493] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.123929 2742 raft_consensus.cc:515] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.124176 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc9ac40dd5b49578945e3e68e235c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.124473 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d664e6b3d8474a03b544efc333d1bab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.125227 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915fd705caa420eb64879ea513b8cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.125435 2742 leader_election.cc:290] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:29.127909 1179 log.cc:1199] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/6689cf0804f946c18ebc555a191549d4
I20251029 08:52:29.128463 1179 ts_tablet_manager.cc:1950] T 6689cf0804f946c18ebc555a191549d4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:29.114117 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.125291 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af913080bfe44a86a76704ffa32cd9e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:29.129086 720 leader_election.cc:343] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.130048 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1030) took 2961 ms. Trace:
I20251029 08:52:29.132066 1179 rpcz_store.cc:276] 1029 08:52:26.168260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.181223 (+ 12963us) service_pool.cc:225] Handling call
1029 08:52:29.052001 (+2870778us) ts_tablet_manager.cc:1151] Deleting tablet 6689cf0804f946c18ebc555a191549d4
1029 08:52:29.052138 (+ 137us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.100175 (+ 48037us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.105678 (+ 5503us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.130027 (+ 24349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2870111}
I20251029 08:52:29.132207 1183 tablet_bootstrap.cc:492] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.132946 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1043) took 2922 ms. Trace:
I20251029 08:52:29.133195 1179 rpcz_store.cc:276] 1029 08:52:26.210039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.210089 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:29.132724 (+2922635us) ts_tablet_manager.cc:1151] Deleting tablet 919e5494b84343639a2b9b81b05f1c2c
1029 08:52:29.132738 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.132925 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2922464}
I20251029 08:52:29.134040 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1045) took 2923 ms. Trace:
I20251029 08:52:29.135855 1179 rpcz_store.cc:276] 1029 08:52:26.210662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.210718 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:29.133766 (+2923048us) ts_tablet_manager.cc:1151] Deleting tablet 3c9cdcec8e494050a475defd0cfd119f
1029 08:52:29.133812 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.134022 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2922856}
I20251029 08:52:29.137758 1183 ts_tablet_manager.cc:1403] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.003s
I20251029 08:52:29.138567 1176 ts_tablet_manager.cc:1916] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.140688 758 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d2df9e1e744ed1a6cf549ab57ed266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:28 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.140214 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1047) took 2813 ms. Trace:
W20251029 08:52:29.152195 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 07bf5d693be240f18bacccc81a6ea5d4 (table default.Table_35877e0584a84248a87841ffcb0cbf94 [id=0e0c97d7ced7490b86425ea2e2e4530c], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 07bf5d693be240f18bacccc81a6ea5d4 consensus state has no leader
I20251029 08:52:29.152307 1183 raft_consensus.cc:359] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.152495 1183 raft_consensus.cc:385] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.152547 1183 raft_consensus.cc:740] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.141911 1179 rpcz_store.cc:276] 1029 08:52:26.326371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.326429 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:29.137991 (+2811562us) ts_tablet_manager.cc:1151] Deleting tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac
1029 08:52:29.138323 (+ 332us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.140196 (+ 1873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2802707}
I20251029 08:52:29.156376 1179 tablet_replica.cc:333] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
W20251029 08:52:29.156656 861 consensus_peers.cc:597] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24834985a71349b9865840e9b5537d72. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.156880 1179 raft_consensus.cc:2243] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.157351 1179 raft_consensus.cc:2272] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.158185 1179 ts_tablet_manager.cc:1916] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.158758 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 6689cf0804f946c18ebc555a191549d4 (table default.Table_a565b344ef8e4b60b07cb259f877df2a [id=bef1fc3338f941f18e6ce2019247b13e]) successfully deleted
I20251029 08:52:29.158457 1183 consensus_queue.cc:260] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.167901 1183 ts_tablet_manager.cc:1434] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251029 08:52:29.169644 1974 ts_tablet_manager.cc:1929] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 08:52:29.170447 2781 log.cc:927] Time spent T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251029 08:52:29.170967 1183 tablet_bootstrap.cc:492] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.111882 1178 raft_consensus.cc:359] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.171165 1178 raft_consensus.cc:385] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.171231 1178 raft_consensus.cc:740] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.171552 1178 consensus_queue.cc:260] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.172144 1178 ts_tablet_manager.cc:1434] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.156s user 0.010s sys 0.001s
I20251029 08:52:29.172456 1178 tablet_bootstrap.cc:492] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.173753 1178 tablet_bootstrap.cc:654] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.173748 1183 tablet_bootstrap.cc:654] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:29.177179 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 919e5494b84343639a2b9b81b05f1c2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 919e5494b84343639a2b9b81b05f1c2c already in progress: creating tablet
W20251029 08:52:29.177392 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3c9cdcec8e494050a475defd0cfd119f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c9cdcec8e494050a475defd0cfd119f already in progress: creating tablet
W20251029 08:52:29.180660 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac already in progress: creating tablet
I20251029 08:52:29.198011 589 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b480e5efd23d4dad948244ad7f7b7ed5
I20251029 08:52:29.199573 1176 ts_tablet_manager.cc:1929] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:29.199689 1176 log.cc:1199] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/c9ed0a270b794e469d2fdbd0ea1d96f8
I20251029 08:52:29.200284 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c9b611ae08a247d3ba02eb177c6b1b12
I20251029 08:52:29.200770 593 catalog_manager.cc:6719] ReplaceTablet: tablet c9b611ae08a247d3ba02eb177c6b1b12 deleted and replaced by tablet 802a005d131f48b8b768f97e57038000
W20251029 08:52:29.201872 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 211f3fde17f64c99a0b80e377c028fb4 consensus state has no leader
I20251029 08:52:29.204126 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 802a005d131f48b8b768f97e57038000) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.207454 1179 ts_tablet_manager.cc:1929] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:29.210251 1176 ts_tablet_manager.cc:1950] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.211261 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1051) took 2955 ms. Trace:
I20251029 08:52:29.211374 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af913080bfe44a86a76704ffa32cd9e1" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.211359 1176 rpcz_store.cc:276] 1029 08:52:26.255902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.256744 (+ 842us) service_pool.cc:225] Handling call
1029 08:52:29.110910 (+2854166us) ts_tablet_manager.cc:1151] Deleting tablet c9ed0a270b794e469d2fdbd0ea1d96f8
1029 08:52:29.110920 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.183053 (+ 72133us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.199456 (+ 16403us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.211235 (+ 11779us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26643,"tablet-delete.queue_time_us":2853992}
I20251029 08:52:29.211795 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1046) took 2989 ms. Trace:
I20251029 08:52:29.211874 1176 rpcz_store.cc:276] 1029 08:52:26.222432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.259216 (+ 36784us) service_pool.cc:225] Handling call
1029 08:52:29.211659 (+2952443us) ts_tablet_manager.cc:1151] Deleting tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac
1029 08:52:29.211672 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.211780 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2951228}
I20251029 08:52:29.212222 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1067) took 2162 ms. Trace:
I20251029 08:52:29.212319 1176 rpcz_store.cc:276] 1029 08:52:27.049427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.049726 (+ 299us) service_pool.cc:225] Handling call
1029 08:52:29.212115 (+2162389us) ts_tablet_manager.cc:1151] Deleting tablet 8c3da4b29e3c4208ae2109789f994c1e
1029 08:52:29.212123 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.212203 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162233}
I20251029 08:52:29.212580 1176 tablet_replica.cc:333] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.212771 1176 raft_consensus.cc:2243] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.212924 1176 raft_consensus.cc:2272] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.213686 750 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 802a005d131f48b8b768f97e57038000) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:29.214989 718 leader_election.cc:343] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.215209 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_b58caa6aed5542948934aeca3e67d32d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]) successfully deleted
W20251029 08:52:29.215687 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac already in progress: creating tablet
W20251029 08:52:29.216218 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8c3da4b29e3c4208ae2109789f994c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3da4b29e3c4208ae2109789f994c1e already in progress: creating tablet
I20251029 08:52:29.216760 886 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b611ae08a247d3ba02eb177c6b1b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 802a005d131f48b8b768f97e57038000) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.217183 718 leader_election.cc:304] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:29.217537 718 raft_consensus.cc:2686] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 08:52:29.218977 2756 tablet.cc:1722] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:29.223176 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b61829fcd59d4ad4ab427270b7089d57
I20251029 08:52:29.225103 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:29.226459 1179 log.cc:1199] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c9ed0a270b794e469d2fdbd0ea1d96f8
I20251029 08:52:29.214365 2755 tablet.cc:1722] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:29.215015 998 consensus_peers.cc:597] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d50e2fdfc0e64516a41c90f5a56f4c89. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.225275 1117 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:29.209712 1183 tablet_bootstrap.cc:492] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.232206 1183 ts_tablet_manager.cc:1403] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20251029 08:52:29.232967 1183 raft_consensus.cc:359] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.233198 1183 raft_consensus.cc:385] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.233284 1183 raft_consensus.cc:740] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.233580 1183 consensus_queue.cc:260] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.234176 1183 ts_tablet_manager.cc:1434] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:29.234467 1183 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.235818 1183 tablet_bootstrap.cc:654] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.238953 1176 ts_tablet_manager.cc:1916] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.242241 1974 log.cc:1199] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:29.236256 1179 ts_tablet_manager.cc:1950] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:29.242663 1974 ts_tablet_manager.cc:1950] T 1dd62833c9d548adabbbf71856a4b181 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:29.243023 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1006) took 4185 ms. Trace:
I20251029 08:52:29.243129 1974 rpcz_store.cc:276] 1029 08:52:25.057524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.058974 (+ 1450us) service_pool.cc:225] Handling call
1029 08:52:28.357765 (+3298791us) ts_tablet_manager.cc:1151] Deleting tablet 1dd62833c9d548adabbbf71856a4b181
1029 08:52:28.357779 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.130167 (+772388us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.169556 (+ 39389us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.243003 (+ 73447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3292566}
I20251029 08:52:29.243875 1974 tablet_replica.cc:333] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:29.244100 1974 raft_consensus.cc:2243] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:29.244474 1974 raft_consensus.cc:2272] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.244803 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 1dd62833c9d548adabbbf71856a4b181 (table default.Table_317871afd272485a9a1789cf2f56a94f [id=d2582a8555724b109414545513ee6253]) successfully deleted
I20251029 08:52:29.245100 1974 ts_tablet_manager.cc:1916] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.245275 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1052) took 2902 ms. Trace:
I20251029 08:52:29.249020 1179 rpcz_store.cc:276] 1029 08:52:26.342662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.348418 (+ 5756us) service_pool.cc:225] Handling call
1029 08:52:29.156319 (+2807901us) ts_tablet_manager.cc:1151] Deleting tablet c9ed0a270b794e469d2fdbd0ea1d96f8
1029 08:52:29.156330 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.204342 (+ 48012us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.207359 (+ 3017us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.245258 (+ 37899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2807715}
I20251029 08:52:29.249774 1179 tablet_replica.cc:333] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:29.250207 1179 raft_consensus.cc:2243] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.250552 883 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 178) took 1372 ms. Trace:
I20251029 08:52:29.250619 1179 raft_consensus.cc:2272] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:29.253904 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70b8c594aa24f37a169d1aa29c2268e consensus state has no leader
I20251029 08:52:29.256269 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 5da61591db06416aaeb23a65165e42ee
W20251029 08:52:29.258246 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 442c37e1ee724b6b96f679f05640baf9 consensus state has no leader
I20251029 08:52:29.259944 1176 ts_tablet_manager.cc:1929] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.261149 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_351d968afa8d4394aae3dfc23405ff0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.261790 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351d968afa8d4394aae3dfc23405ff0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.266616 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cddcc5d90ac4457fbbe4489988f438f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.267205 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cddcc5d90ac4457fbbe4489988f438f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.268720 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e52a0b50fc624679bc5d91a8cf424a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.269117 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e52a0b50fc624679bc5d91a8cf424a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.270274 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_edb90c7ab4f746b6b8fd878de2a107a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.270653 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb90c7ab4f746b6b8fd878de2a107a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.272322 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_073dbf71123c4416bb365911b2d93ad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.272805 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073dbf71123c4416bb365911b2d93ad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.273277 1176 log.cc:1199] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/af913080bfe44a86a76704ffa32cd9e1
I20251029 08:52:29.273648 1176 ts_tablet_manager.cc:1950] T af913080bfe44a86a76704ffa32cd9e1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.274014 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a050762711c3496094db8f582859d8b3" } modify_external_catalogs: true
I20251029 08:52:29.274471 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1070) took 2215 ms. Trace:
I20251029 08:52:29.274484 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a050762711c3496094db8f582859d8b3" } modify_external_catalogs: true
I20251029 08:52:29.274554 1176 rpcz_store.cc:276] 1029 08:52:27.058510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.058561 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:29.212540 (+2153979us) ts_tablet_manager.cc:1151] Deleting tablet af913080bfe44a86a76704ffa32cd9e1
1029 08:52:29.212548 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.256942 (+ 44394us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.259860 (+ 2918us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.274447 (+ 14587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153773}
I20251029 08:52:29.274942 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1068) took 2217 ms. Trace:
I20251029 08:52:29.275022 1176 rpcz_store.cc:276] 1029 08:52:27.057639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.069258 (+ 11619us) service_pool.cc:225] Handling call
1029 08:52:29.274820 (+2205562us) ts_tablet_manager.cc:1151] Deleting tablet d965d5a6b1324b00ad68b28d76796e80
1029 08:52:29.274830 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.274926 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2205281}
I20251029 08:52:29.275272 1176 tablet_replica.cc:333] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.275477 1176 raft_consensus.cc:2243] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.275612 1176 raft_consensus.cc:2272] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.276934 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_b58caa6aed5542948934aeca3e67d32d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]) successfully deleted
I20251029 08:52:29.277810 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b61829fcd59d4ad4ab427270b7089d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.278054 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet af913080bfe44a86a76704ffa32cd9e1 (table default.Table_7fd53f3b59d14ad39715bc62e513ab79 [id=88bde60591764b37b719bdeea15ef0cd]) successfully deleted
W20251029 08:52:29.278314 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d965d5a6b1324b00ad68b28d76796e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d965d5a6b1324b00ad68b28d76796e80 already in progress: creating tablet
I20251029 08:52:29.283799 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_30db36162b644be4af80133b2d49d317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.284325 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30db36162b644be4af80133b2d49d317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.286454 1176 ts_tablet_manager.cc:1916] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.305721 1176 ts_tablet_manager.cc:1929] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.305859 1176 log.cc:1199] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3cd2e9a95b6c40be9e17397a313d249b
I20251029 08:52:29.306159 1176 ts_tablet_manager.cc:1950] T 3cd2e9a95b6c40be9e17397a313d249b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.309018 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1072) took 2237 ms. Trace:
I20251029 08:52:29.309114 1176 rpcz_store.cc:276] 1029 08:52:27.071513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.071623 (+ 110us) service_pool.cc:225] Handling call
1029 08:52:29.275224 (+2203601us) ts_tablet_manager.cc:1151] Deleting tablet 3cd2e9a95b6c40be9e17397a313d249b
1029 08:52:29.275233 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.298421 (+ 23188us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.305609 (+ 7188us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.308995 (+ 3386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203424}
I20251029 08:52:29.309561 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 3cd2e9a95b6c40be9e17397a313d249b (table default.Table_79896e97f2934c50a068bf4beaf34966 [id=c6901c2dab494740be7681afa3f4aa9f]) successfully deleted
I20251029 08:52:29.309646 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b0e8b56db4984d038e0b83a4948bd8b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.310072 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 89373.8us
W20251029 08:52:29.310199 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0e8b56db4984d038e0b83a4948bd8b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.311357 1178 tablet_bootstrap.cc:492] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.311501 1178 ts_tablet_manager.cc:1403] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.139s user 0.002s sys 0.000s
I20251029 08:52:29.312240 1178 raft_consensus.cc:359] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.312403 1178 raft_consensus.cc:385] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.312448 1178 raft_consensus.cc:740] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.312669 1178 consensus_queue.cc:260] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.313143 1178 ts_tablet_manager.cc:1434] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:29.313964 1179 ts_tablet_manager.cc:1916] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.314291 1178 tablet_bootstrap.cc:492] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.315821 1178 tablet_bootstrap.cc:654] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.276952 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b61829fcd59d4ad4ab427270b7089d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.277637 2707 consensus_queue.cc:1048] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 08:52:29.293627 2485 raft_consensus.cc:493] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.324695 2485 raft_consensus.cc:515] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.325289 2485 leader_election.cc:290] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.325641 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f68e8291cd64391bdb236f948dd338b" } new_table_name: "default.Table_9e51faa7dd614a80881dbf7b6762b2c7" modify_external_catalogs: true
I20251029 08:52:29.326526 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:29.326690 773 raft_consensus.cc:2468] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:29.327733 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1d7a8b08b5104bb2afd66ad8dec95210" } new_table_name: "default.Table_e006f70bed4248f890eb94f10f2d90f8" modify_external_catalogs: true
I20251029 08:52:29.329198 998 leader_election.cc:304] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:29.329916 2485 raft_consensus.cc:2804] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:29.330011 2485 raft_consensus.cc:493] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:29.330067 2485 raft_consensus.cc:3060] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.255206 2451 raft_consensus.cc:493] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.330685 2451 raft_consensus.cc:515] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.331216 2451 leader_election.cc:290] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.332000 883 rpcz_store.cc:276] 1029 08:52:27.878476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.878531 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:27.984686 (+106155us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x6160000b3e54. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508d2e0200 00007f509119066f 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8 0000000000a8867c
1029 08:52:29.250504 (+1265818us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11343616}
I20251029 08:52:29.261407 2554 consensus_queue.cc:1048] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:29.334630 2485 raft_consensus.cc:515] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.277235 886 tablet_service.cc:1552] Processing DeleteTablet for tablet b61829fcd59d4ad4ab427270b7089d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.335438 1176 tablet_replica.cc:333] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.336246 1176 raft_consensus.cc:2243] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.336398 1176 raft_consensus.cc:2272] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.336884 1176 ts_tablet_manager.cc:1916] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.339522 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:29.339686 773 raft_consensus.cc:3060] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.340497 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:29.341018 996 leader_election.cc:343] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.342443 773 raft_consensus.cc:2468] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:29.343742 998 leader_election.cc:304] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.344041 2359 raft_consensus.cc:2804] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.344135 2359 raft_consensus.cc:697] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:29.344358 2359 consensus_queue.cc:237] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.346076 2485 leader_election.cc:290] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.354418 884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 177) took 1477 ms. Trace:
I20251029 08:52:29.354548 884 rpcz_store.cc:276] 1029 08:52:27.876550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.876799 (+ 249us) service_pool.cc:225] Handling call
1029 08:52:27.984386 (+107587us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x6160000b3e54. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508d2e0200 00007f509119066f 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8 0000000000a8867c
1029 08:52:28.384549 (+400163us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x6180000f8c8c. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508e86d44e 00007f508e668ac1 00007f5091190478 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8
1029 08:52:29.194640 (+810091us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x618000104c8c. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508e86d44e 00007f508e668ac1 00007f5091190478 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8
1029 08:52:29.354367 (+159727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120727424}
I20251029 08:52:29.300766 1177 tablet_bootstrap.cc:492] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
W20251029 08:52:29.321058 861 consensus_peers.cc:597] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9ed0a270b794e469d2fdbd0ea1d96f8. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.357298 1177 ts_tablet_manager.cc:1403] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.895s user 0.014s sys 0.031s
I20251029 08:52:29.326033 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0fa089fce564af4b03aa323cec6cbf6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.358289 1177 raft_consensus.cc:359] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.358448 1177 raft_consensus.cc:385] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.358507 1177 raft_consensus.cc:740] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.358701 1177 consensus_queue.cc:260] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:29.358754 996 leader_election.cc:343] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.359288 861 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:29.359776 1176 ts_tablet_manager.cc:1929] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.359884 1176 log.cc:1199] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/391977048e5f4f0b8f62e4d5e3d2946b
I20251029 08:52:29.360194 1176 ts_tablet_manager.cc:1950] T 391977048e5f4f0b8f62e4d5e3d2946b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.330538 882 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 176) took 1595 ms. Trace:
I20251029 08:52:29.360327 882 rpcz_store.cc:276] 1029 08:52:27.734854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.734912 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:28.518195 (+783283us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x616000222754. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508d2e0200 00007f509119066f 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8 0000000000a8867c
1029 08:52:29.179419 (+661224us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x618000104c8c. stack: 00007f50814469f6 00007f50814467b0 00007f50804223ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f508e86d44e 00007f508e668ac1 00007f5091190478 00007f508b5b6c36 00007f508b5b6965 00007f50883e527e 00007f50883e415e 00007f50883ea395 00007f50883eacbf 00007f50883eaae8
1029 08:52:29.330488 (+151069us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275126912}
I20251029 08:52:29.360971 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1073) took 2289 ms. Trace:
I20251029 08:52:29.361088 1176 rpcz_store.cc:276] 1029 08:52:27.071945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.071987 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:29.335383 (+2263396us) ts_tablet_manager.cc:1151] Deleting tablet 391977048e5f4f0b8f62e4d5e3d2946b
1029 08:52:29.335394 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.340213 (+ 4819us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.359649 (+ 19436us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.360946 (+ 1297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263228}
I20251029 08:52:29.363078 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 391977048e5f4f0b8f62e4d5e3d2946b (table default.Table_cf4be2bf29e44a4bbb908e06223cc48d [id=48afa12a4fd24dc0918cd31e4da595b5]) successfully deleted
I20251029 08:52:29.366092 1177 ts_tablet_manager.cc:1434] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:29.366518 1177 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.369160 972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
I20251029 08:52:29.372170 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e65b3b1c8d14743aca1465d95ace95f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.373445 2359 raft_consensus.cc:493] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.373654 2359 raft_consensus.cc:515] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.374747 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1074) took 2302 ms. Trace:
I20251029 08:52:29.374827 2359 leader_election.cc:290] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.374858 1176 rpcz_store.cc:276] 1029 08:52:27.072307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.072348 (+ 41us) service_pool.cc:225] Handling call
1029 08:52:29.374605 (+2302257us) ts_tablet_manager.cc:1151] Deleting tablet 58932f38c0d64aec9a02238eaea56a25
1029 08:52:29.374617 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.374726 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2302124}
I20251029 08:52:29.374995 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09968d93203140fa9201928c87702ee0" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.375228 1065 raft_consensus.cc:2468] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:29.375391 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 58932f38c0d64aec9a02238eaea56a25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58932f38c0d64aec9a02238eaea56a25 already in progress: creating tablet
I20251029 08:52:29.375633 861 leader_election.cc:304] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:29.375672 996 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.375880 998 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09968d93203140fa9201928c87702ee0
I20251029 08:52:29.375993 998 leader_election.cc:304] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.376008 2554 raft_consensus.cc:2804] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:29.376154 2554 raft_consensus.cc:493] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:29.376219 2554 raft_consensus.cc:3060] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.376375 2359 raft_consensus.cc:2749] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.376650 1176 tablet_replica.cc:333] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.376857 1176 raft_consensus.cc:2243] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.377012 1176 raft_consensus.cc:2272] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.378934 2554 raft_consensus.cc:515] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.379756 2554 leader_election.cc:290] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.248840 1183 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.380314 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e65b3b1c8d14743aca1465d95ace95f" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
W20251029 08:52:29.380544 861 leader_election.cc:343] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:29.380645 1067 raft_consensus.cc:3060] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.382041 1177 tablet_bootstrap.cc:654] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.383330 1067 raft_consensus.cc:2468] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:29.383692 861 leader_election.cc:304] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:29.384173 2554 raft_consensus.cc:2804] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.384382 2554 raft_consensus.cc:697] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:29.384558 1176 ts_tablet_manager.cc:1916] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.384701 2554 consensus_queue.cc:237] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.384857 1178 tablet_bootstrap.cc:492] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.384968 1178 ts_tablet_manager.cc:1403] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.000s
I20251029 08:52:29.387761 1974 ts_tablet_manager.cc:1929] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:29.387857 1974 log.cc:1199] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:29.380649 1183 ts_tablet_manager.cc:1403] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.146s user 0.002s sys 0.001s
I20251029 08:52:29.396914 1183 raft_consensus.cc:359] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.397092 1177 tablet_bootstrap.cc:492] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:29.397279 1177 ts_tablet_manager.cc:1403] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20251029 08:52:29.397940 1177 raft_consensus.cc:359] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.398154 1177 raft_consensus.cc:385] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.398293 1177 raft_consensus.cc:740] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.398676 1177 consensus_queue.cc:260] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.399417 1183 raft_consensus.cc:385] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.399503 1183 raft_consensus.cc:740] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.400596 1176 ts_tablet_manager.cc:1929] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.402558 1176 log.cc:1199] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e0fa089fce564af4b03aa323cec6cbf6
I20251029 08:52:29.402877 1176 ts_tablet_manager.cc:1950] T e0fa089fce564af4b03aa323cec6cbf6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.402786 1183 consensus_queue.cc:260] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.403319 1183 ts_tablet_manager.cc:1434] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:29.406713 594 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ec00cd64dc247da8442b6ba197f370a from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.385730 1178 raft_consensus.cc:359] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.407323 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64051e12990148d9aad8a1bd220d49e8" } modify_external_catalogs: true
I20251029 08:52:29.407440 1178 raft_consensus.cc:385] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.407464 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_64051e12990148d9aad8a1bd220d49e8" } modify_external_catalogs: true
I20251029 08:52:29.407521 1178 raft_consensus.cc:740] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.407528 1177 ts_tablet_manager.cc:1434] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:29.407747 1178 consensus_queue.cc:260] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.408272 1178 ts_tablet_manager.cc:1434] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251029 08:52:29.409015 1177 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.410673 1177 tablet_bootstrap.cc:654] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.411211 1183 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.403638 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1071) took 2332 ms. Trace:
I20251029 08:52:29.411885 1176 rpcz_store.cc:276] 1029 08:52:27.071410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.086105 (+ 14695us) service_pool.cc:225] Handling call
1029 08:52:29.376600 (+2290495us) ts_tablet_manager.cc:1151] Deleting tablet e0fa089fce564af4b03aa323cec6cbf6
1029 08:52:29.376611 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.397465 (+ 20854us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.400500 (+ 3035us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.403615 (+ 3115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2290012}
I20251029 08:52:29.412389 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e0fa089fce564af4b03aa323cec6cbf6 (table default.Table_8d4ea5c42591424eb1be9e9e3981cc89 [id=1c0e2e0996134fe1811cb2e300a861ac]) successfully deleted
I20251029 08:52:29.412832 1183 tablet_bootstrap.cc:654] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.420859 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ea2dc34a3418478cbf34a81ec9dd33e1" } modify_external_catalogs: true
I20251029 08:52:29.421033 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ea2dc34a3418478cbf34a81ec9dd33e1" } modify_external_catalogs: true
I20251029 08:52:29.422102 1183 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.422230 1183 ts_tablet_manager.cc:1403] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20251029 08:52:29.422953 1183 raft_consensus.cc:359] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.423123 1183 raft_consensus.cc:385] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.423188 1183 raft_consensus.cc:740] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.426020 1974 ts_tablet_manager.cc:1950] T aa411391c63146e7acb8b4c6edbe12a0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:29.426920 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1007) took 4369 ms. Trace:
I20251029 08:52:29.427059 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_83f92101d57a49da80bdc2552b39a1ba" } new_table_name: "default.Table_a5ebbce3f0ef404699dae9a0c1c670c3" modify_external_catalogs: true
I20251029 08:52:29.427034 1974 rpcz_store.cc:276] 1029 08:52:25.057618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.059174 (+ 1556us) service_pool.cc:225] Handling call
1029 08:52:29.243825 (+4184651us) ts_tablet_manager.cc:1151] Deleting tablet aa411391c63146e7acb8b4c6edbe12a0
1029 08:52:29.243836 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.363893 (+120057us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.387656 (+ 23763us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.426896 (+ 39240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4178738}
I20251029 08:52:29.427582 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1013) took 4295 ms. Trace:
I20251029 08:52:29.427654 1974 rpcz_store.cc:276] 1029 08:52:25.132512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.132568 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:29.427440 (+4294872us) ts_tablet_manager.cc:1151] Deleting tablet 8b1af4bcd7af49a0b8a419298ab37ea9
1029 08:52:29.427451 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.427564 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4294679}
I20251029 08:52:29.428117 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1012) took 4295 ms. Trace:
I20251029 08:52:29.428102 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet aa411391c63146e7acb8b4c6edbe12a0 (table default.Table_dd22da054ccc4c1f87f65c41649bd1a5 [id=6f1a696dded848c89a9160362541702a]) successfully deleted
I20251029 08:52:29.428174 1974 rpcz_store.cc:276] 1029 08:52:25.132364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.136646 (+ 4282us) service_pool.cc:225] Handling call
1029 08:52:29.427996 (+4291350us) ts_tablet_manager.cc:1151] Deleting tablet a2c13db40d854269b13340aa6acef83e
1029 08:52:29.428006 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.428101 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4291150}
W20251029 08:52:29.428346 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8b1af4bcd7af49a0b8a419298ab37ea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b1af4bcd7af49a0b8a419298ab37ea9 already in progress: creating tablet
W20251029 08:52:29.428560 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a2c13db40d854269b13340aa6acef83e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c13db40d854269b13340aa6acef83e already in progress: creating tablet
I20251029 08:52:29.428575 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1018) took 4212 ms. Trace:
I20251029 08:52:29.428630 1974 rpcz_store.cc:276] 1029 08:52:25.215775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.216431 (+ 656us) service_pool.cc:225] Handling call
1029 08:52:29.428465 (+4212034us) ts_tablet_manager.cc:1151] Deleting tablet f062556d67a148d19d242b8d99cfdff9
1029 08:52:29.428473 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.428559 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4148775}
I20251029 08:52:29.428891 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7831bec582164564bcbadb55b6110645"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.429414 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7831bec582164564bcbadb55b6110645 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:29.429541 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f062556d67a148d19d242b8d99cfdff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f062556d67a148d19d242b8d99cfdff9 already in progress: creating tablet
I20251029 08:52:29.429849 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1023) took 3989 ms. Trace:
I20251029 08:52:29.429917 1974 rpcz_store.cc:276] 1029 08:52:25.439913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.439981 (+ 68us) service_pool.cc:225] Handling call
1029 08:52:29.429752 (+3989771us) ts_tablet_manager.cc:1151] Deleting tablet d0d00a6b772d4034ac7fa764884c7081
1029 08:52:29.429762 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.429833 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3988782}
W20251029 08:52:29.430246 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d0d00a6b772d4034ac7fa764884c7081 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0d00a6b772d4034ac7fa764884c7081 already in progress: creating tablet
I20251029 08:52:29.430519 1974 tablet_replica.cc:333] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:29.430689 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 850aee25ae1940bcbe6677dcea2e7b1d consensus state has no leader
I20251029 08:52:29.430720 1974 raft_consensus.cc:2243] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:29.430933 592 master_service.cc:954] ReplaceTablet: received request to replace tablet 42de32d87ebb451090e0310e36e64277 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.431026 1974 raft_consensus.cc:2272] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.431579 1974 ts_tablet_manager.cc:1916] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.431567 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_620d0574d4154375bbf3774c5ad37097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.432166 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_620d0574d4154375bbf3774c5ad37097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.433537 1178 tablet_bootstrap.cc:492] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.435252 1178 tablet_bootstrap.cc:654] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.435777 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d07c3391755f4f82bd1579cd0d3b2c49" } modify_external_catalogs: true
I20251029 08:52:29.435966 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d07c3391755f4f82bd1579cd0d3b2c49" } modify_external_catalogs: true
I20251029 08:52:29.438557 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1082) took 2200 ms. Trace:
I20251029 08:52:29.438694 1176 rpcz_store.cc:276] 1029 08:52:27.238079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.238137 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:29.413008 (+2174871us) ts_tablet_manager.cc:1151] Deleting tablet 2fad7549758649229ad771de926aaf15
1029 08:52:29.438426 (+ 25418us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.438537 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174678}
W20251029 08:52:29.439154 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 2fad7549758649229ad771de926aaf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fad7549758649229ad771de926aaf15 already in progress: creating tablet
I20251029 08:52:29.439563 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1081) took 2202 ms. Trace:
I20251029 08:52:29.439714 1176 rpcz_store.cc:276] 1029 08:52:27.237394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.238677 (+ 1283us) service_pool.cc:225] Handling call
1029 08:52:29.439458 (+2200781us) ts_tablet_manager.cc:1151] Deleting tablet bd357a723a1d4bc1aa4b480c8a29ef03
1029 08:52:29.439467 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.439546 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2200515}
W20251029 08:52:29.440603 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bd357a723a1d4bc1aa4b480c8a29ef03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd357a723a1d4bc1aa4b480c8a29ef03 already in progress: creating tablet
I20251029 08:52:29.441109 1176 tablet_replica.cc:333] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.441335 1176 raft_consensus.cc:2243] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.441541 1176 raft_consensus.cc:2272] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.442174 1176 ts_tablet_manager.cc:1916] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.442556 1179 ts_tablet_manager.cc:1929] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.461241 1176 ts_tablet_manager.cc:1929] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.461445 1176 log.cc:1199] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9907d044daa145168417371c2bd66544
I20251029 08:52:29.461845 1176 ts_tablet_manager.cc:1950] T 9907d044daa145168417371c2bd66544 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.463326 2764 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:29.467795 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1084) took 2186 ms. Trace:
I20251029 08:52:29.467902 1176 rpcz_store.cc:276] 1029 08:52:27.281238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.281290 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:29.440902 (+2159612us) ts_tablet_manager.cc:1151] Deleting tablet 9907d044daa145168417371c2bd66544
1029 08:52:29.441060 (+ 158us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.450949 (+ 9889us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.461083 (+ 10134us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.467768 (+ 6685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159409}
W20251029 08:52:29.468605 861 consensus_peers.cc:597] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19fdc381a0d949b0b93cef399daa3021. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.468695 1176 tablet_replica.cc:333] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.469563 1176 raft_consensus.cc:2243] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.468394 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9907d044daa145168417371c2bd66544 (table default.Table_325e6358252a40b0bdd59c1a0664ed7e [id=3bb1d67e45ea4d7fbc5f901e60cf7083]) successfully deleted
I20251029 08:52:29.472411 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f41fe46951a049778b48cb9eb97e3fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.470114 1176 raft_consensus.cc:2272] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.474423 1176 ts_tablet_manager.cc:1916] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:29.475348 861 consensus_peers.cc:597] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3955ff79faf42bb82f764aae79b85d9. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.477077 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d782dc9366fe4e2bbc42fe62f589e7b5" } new_table_name: "default.Table_d8be37074ea440ef9fbf1e9d596dbf39" modify_external_catalogs: true
W20251029 08:52:29.488425 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 56122.1us
I20251029 08:52:29.490821 2756 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 08:52:29.493126 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f41fe46951a049778b48cb9eb97e3fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.496142 1179 log.cc:1199] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9adbcf3f12104fb5bfd1d9dd5c946c54
I20251029 08:52:29.496532 1179 ts_tablet_manager.cc:1950] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:29.464002 2755 tablet.cc:1722] T ca5bfc2ee3c6491eb6857f6b9855daf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:29.465898 1183 consensus_queue.cc:260] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.497382 1183 ts_tablet_manager.cc:1434] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.075s user 0.001s sys 0.001s
W20251029 08:52:29.497522 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f91eb858ae344549a78357e04f7e938 consensus state has no leader
I20251029 08:52:29.497674 1183 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.497923 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b864bd5ff3e341938015d4bae27df6d4" } modify_external_catalogs: true
I20251029 08:52:29.498070 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b864bd5ff3e341938015d4bae27df6d4" } modify_external_catalogs: true
I20251029 08:52:29.499300 1183 tablet_bootstrap.cc:654] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.500361 1176 ts_tablet_manager.cc:1929] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.500471 1176 log.cc:1199] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/dbf9e36c47be4509a998be8e7138475b
I20251029 08:52:29.504483 1176 ts_tablet_manager.cc:1950] T dbf9e36c47be4509a998be8e7138475b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.505215 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1085) took 2223 ms. Trace:
I20251029 08:52:29.506906 2786 raft_consensus.cc:493] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.505297 1176 rpcz_store.cc:276] 1029 08:52:27.281696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.281740 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:29.468591 (+2186851us) ts_tablet_manager.cc:1151] Deleting tablet dbf9e36c47be4509a998be8e7138475b
1029 08:52:29.468602 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.486581 (+ 17979us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.500246 (+ 13665us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.505194 (+ 4948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2186668}
I20251029 08:52:29.507041 2786 raft_consensus.cc:515] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.507321 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet dbf9e36c47be4509a998be8e7138475b (table default.Table_47b07a98fcc34ec2a2784a43b313f904 [id=0265c5ac448a46a49a1bfcca4a305645]) successfully deleted
I20251029 08:52:29.507680 2786 leader_election.cc:290] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:29.507694 1176 tablet_replica.cc:333] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.508091 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.508292 1176 raft_consensus.cc:2243] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.508353 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.508471 1176 raft_consensus.cc:2272] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:29.508949 718 leader_election.cc:343] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.509199 720 leader_election.cc:343] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.509292 720 leader_election.cc:304] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.510002 1176 ts_tablet_manager.cc:1916] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.510452 2786 raft_consensus.cc:2749] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.511278 1178 tablet_bootstrap.cc:492] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.511404 1178 ts_tablet_manager.cc:1403] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20251029 08:52:29.512234 1178 raft_consensus.cc:359] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.512396 1178 raft_consensus.cc:385] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.512447 1178 raft_consensus.cc:740] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.512652 1178 consensus_queue.cc:260] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.513118 1178 ts_tablet_manager.cc:1434] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:29.513185 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_fef23ec2ce374eadbd727deb26c45850" } new_table_name: "default.Table_18c407e625464254aa01f6cc9e12144f" modify_external_catalogs: true
I20251029 08:52:29.513448 1178 tablet_bootstrap.cc:492] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.513247 1177 tablet_bootstrap.cc:492] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:29.515367 1177 ts_tablet_manager.cc:1403] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.106s user 0.002s sys 0.003s
I20251029 08:52:29.516100 1177 raft_consensus.cc:359] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.516481 1178 tablet_bootstrap.cc:654] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.516597 1177 raft_consensus.cc:385] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.520545 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1048) took 3193 ms. Trace:
I20251029 08:52:29.520933 1183 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.521080 1183 ts_tablet_manager.cc:1403] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20251029 08:52:29.521857 1183 raft_consensus.cc:359] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.522034 1183 raft_consensus.cc:385] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.522083 1183 raft_consensus.cc:740] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.524682 2786 raft_consensus.cc:493] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.524837 2786 raft_consensus.cc:515] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.524827 1183 consensus_queue.cc:260] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.525424 2786 leader_election.cc:290] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:29.526648 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85df43bf781c421688f163331de0eb2b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.527678 1183 ts_tablet_manager.cc:1434] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 08:52:29.527526 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85df43bf781c421688f163331de0eb2b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:29.527913 720 leader_election.cc:343] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.528203 718 leader_election.cc:343] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.528308 718 leader_election.cc:304] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.528707 2786 raft_consensus.cc:2749] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.528860 1177 raft_consensus.cc:740] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.520871 1179 rpcz_store.cc:276] 1029 08:52:26.326645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.330258 (+ 3613us) service_pool.cc:225] Handling call
1029 08:52:29.249729 (+2919471us) ts_tablet_manager.cc:1151] Deleting tablet 9adbcf3f12104fb5bfd1d9dd5c946c54
1029 08:52:29.249738 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.439094 (+189356us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.442447 (+ 3353us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.520516 (+ 78069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2822914}
I20251029 08:52:29.529129 1177 consensus_queue.cc:260] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.529749 1177 ts_tablet_manager.cc:1434] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251029 08:52:29.530043 1177 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.530278 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:29.531500 1177 tablet_bootstrap.cc:654] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.536921 1176 ts_tablet_manager.cc:1929] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.540480 1974 ts_tablet_manager.cc:1929] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:29.540625 1974 log.cc:1199] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:29.540992 1974 ts_tablet_manager.cc:1950] T 61b07eb4befa41f3a1b5f2e9ebc2277c P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:29.541814 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1027) took 4073 ms. Trace:
I20251029 08:52:29.541920 1974 rpcz_store.cc:276] 1029 08:52:25.468798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.468850 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:29.430462 (+3961612us) ts_tablet_manager.cc:1151] Deleting tablet 61b07eb4befa41f3a1b5f2e9ebc2277c
1029 08:52:29.430471 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.502971 (+ 72500us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.540366 (+ 37395us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.541793 (+ 1427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3961452}
I20251029 08:52:29.542384 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 61b07eb4befa41f3a1b5f2e9ebc2277c (table default.Table_bce9df9d85294f4fb7c3b786993463ca [id=821160def5624241afdce0f734ef2411]) successfully deleted
I20251029 08:52:29.562047 1183 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.563071 1176 log.cc:1199] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/eeabf8f23751447aa9af02363c40361d
I20251029 08:52:29.563823 1176 ts_tablet_manager.cc:1950] T eeabf8f23751447aa9af02363c40361d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.564440 1183 tablet_bootstrap.cc:654] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.566462 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1050) took 3239 ms. Trace:
I20251029 08:52:29.566603 1179 rpcz_store.cc:276] 1029 08:52:26.326956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.470470 (+143514us) service_pool.cc:225] Handling call
1029 08:52:29.529161 (+3058691us) ts_tablet_manager.cc:1151] Deleting tablet 59ad04f57c29407b9d2420520c095a28
1029 08:52:29.566268 (+ 37107us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.566442 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3058484}
W20251029 08:52:29.566828 861 consensus_peers.cc:597] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa0ff14e19b4ad7a8b33796481f0f3f. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:29.567137 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 59ad04f57c29407b9d2420520c095a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ad04f57c29407b9d2420520c095a28 already in progress: creating tablet
I20251029 08:52:29.567533 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1068) took 2518 ms. Trace:
I20251029 08:52:29.567647 1179 rpcz_store.cc:276] 1029 08:52:27.049223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.049492 (+ 269us) service_pool.cc:225] Handling call
1029 08:52:29.567410 (+2517918us) ts_tablet_manager.cc:1151] Deleting tablet 8c3da4b29e3c4208ae2109789f994c1e
1029 08:52:29.567420 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.567515 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2517730}
W20251029 08:52:29.568079 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8c3da4b29e3c4208ae2109789f994c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3da4b29e3c4208ae2109789f994c1e already in progress: creating tablet
I20251029 08:52:29.568447 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1070) took 2510 ms. Trace:
I20251029 08:52:29.568557 1179 rpcz_store.cc:276] 1029 08:52:27.057646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.057700 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:29.568328 (+2510628us) ts_tablet_manager.cc:1151] Deleting tablet d965d5a6b1324b00ad68b28d76796e80
1029 08:52:29.568337 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.568431 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2510461}
I20251029 08:52:29.569597 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1086) took 2287 ms. Trace:
W20251029 08:52:29.571533 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d965d5a6b1324b00ad68b28d76796e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d965d5a6b1324b00ad68b28d76796e80 already in progress: creating tablet
I20251029 08:52:29.571884 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1072) took 2500 ms. Trace:
I20251029 08:52:29.571993 1179 rpcz_store.cc:276] 1029 08:52:27.071771 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.071833 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:29.571783 (+2499950us) ts_tablet_manager.cc:1151] Deleting tablet 58932f38c0d64aec9a02238eaea56a25
1029 08:52:29.571792 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.571870 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2499786}
W20251029 08:52:29.572425 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 58932f38c0d64aec9a02238eaea56a25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58932f38c0d64aec9a02238eaea56a25 already in progress: creating tablet
I20251029 08:52:29.572880 1179 tablet_replica.cc:333] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:29.573099 1179 raft_consensus.cc:2243] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.573292 1179 raft_consensus.cc:2272] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.574076 1179 ts_tablet_manager.cc:1916] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.569725 1176 rpcz_store.cc:276] 1029 08:52:27.282067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.282110 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:29.507613 (+2225503us) ts_tablet_manager.cc:1151] Deleting tablet eeabf8f23751447aa9af02363c40361d
1029 08:52:29.507628 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.531124 (+ 23496us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.536799 (+ 5675us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.569566 (+ 32767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225340}
I20251029 08:52:29.576777 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet eeabf8f23751447aa9af02363c40361d (table default.Table_5fefde56f4da40239f0ccb837befc3db [id=ea6733c09e8f48d2b64292168b11d69a]) successfully deleted
W20251029 08:52:29.577544 861 consensus_peers.cc:597] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9ed0a270b794e469d2fdbd0ea1d96f8. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.576920 1176 tablet_replica.cc:333] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.580407 1176 raft_consensus.cc:2243] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.580657 1176 raft_consensus.cc:2272] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.581275 1176 ts_tablet_manager.cc:1916] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.581833 1178 tablet_bootstrap.cc:492] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.582001 1178 ts_tablet_manager.cc:1403] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251029 08:52:29.582759 1178 raft_consensus.cc:359] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.582950 1178 raft_consensus.cc:385] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.583035 1178 raft_consensus.cc:740] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.583303 1178 consensus_queue.cc:260] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.583870 1178 ts_tablet_manager.cc:1434] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:29.585014 1177 tablet_bootstrap.cc:492] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:29.585569 1177 ts_tablet_manager.cc:1403] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.015s
I20251029 08:52:29.586457 1177 raft_consensus.cc:359] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.586690 1177 raft_consensus.cc:385] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.586786 1177 raft_consensus.cc:740] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.587015 1177 consensus_queue.cc:260] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.587543 1177 ts_tablet_manager.cc:1434] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:29.587836 1177 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.589442 1183 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.589569 1183 ts_tablet_manager.cc:1403] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.009s
I20251029 08:52:29.590265 1183 raft_consensus.cc:359] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.590428 1183 raft_consensus.cc:385] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.590492 1183 raft_consensus.cc:740] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.590674 1178 tablet_bootstrap.cc:492] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.590651 1183 consensus_queue.cc:260] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.591078 1183 ts_tablet_manager.cc:1434] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:29.591579 1183 tablet_bootstrap.cc:492] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.593775 1183 tablet_bootstrap.cc:654] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.603960 1178 tablet_bootstrap.cc:654] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.613890 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1032) took 3945 ms. Trace:
I20251029 08:52:29.614073 1974 rpcz_store.cc:276] 1029 08:52:25.668741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.668804 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:29.613785 (+3944981us) ts_tablet_manager.cc:1151] Deleting tablet a2b7853b85d24a04b8b11caaa4fe8c4d
1029 08:52:29.613797 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.613868 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3944794}
W20251029 08:52:29.614554 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet a2b7853b85d24a04b8b11caaa4fe8c4d because the tablet was not found. No further retry: Not found: Tablet not found: a2b7853b85d24a04b8b11caaa4fe8c4d
I20251029 08:52:29.614909 1974 tablet_replica.cc:333] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:29.615147 1974 raft_consensus.cc:2243] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:29.615571 1974 raft_consensus.cc:2272] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.616189 1974 ts_tablet_manager.cc:1916] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.621697 1176 ts_tablet_manager.cc:1929] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.621811 1176 log.cc:1199] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/65b20d53d6214bdc81ca205dbf922d7b
I20251029 08:52:29.622639 1177 tablet_bootstrap.cc:654] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.623850 1179 ts_tablet_manager.cc:1929] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.626634 1176 ts_tablet_manager.cc:1950] T 65b20d53d6214bdc81ca205dbf922d7b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.627355 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1087) took 2344 ms. Trace:
I20251029 08:52:29.628353 2295 consensus_queue.cc:1048] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:29.627439 1176 rpcz_store.cc:276] 1029 08:52:27.282457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.282499 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:29.576854 (+2294355us) ts_tablet_manager.cc:1151] Deleting tablet 65b20d53d6214bdc81ca205dbf922d7b
1029 08:52:29.576869 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.607904 (+ 31035us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.621585 (+ 13681us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.627336 (+ 5751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294198}
I20251029 08:52:29.635457 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 65b20d53d6214bdc81ca205dbf922d7b (table default.Table_9c4c2744d70f4d1a9e52b9604edcb436 [id=5762d46725ec489d92cc4cb914fe070b]) successfully deleted
I20251029 08:52:29.636790 1176 tablet_replica.cc:333] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.638967 1176 raft_consensus.cc:2243] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.639158 1176 raft_consensus.cc:2272] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.639734 1176 ts_tablet_manager.cc:1916] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.644912 2797 raft_consensus.cc:493] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.651052 2797 raft_consensus.cc:515] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.651319 1179 log.cc:1199] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/af913080bfe44a86a76704ffa32cd9e1
I20251029 08:52:29.651774 2797 leader_election.cc:290] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.651811 1179 ts_tablet_manager.cc:1950] T af913080bfe44a86a76704ffa32cd9e1 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:29.655736 1178 tablet_bootstrap.cc:492] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.655964 1178 ts_tablet_manager.cc:1403] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.002s
I20251029 08:52:29.657227 1177 tablet_bootstrap.cc:492] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:29.657375 1177 ts_tablet_manager.cc:1403] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251029 08:52:29.657405 1178 raft_consensus.cc:359] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.657698 1178 raft_consensus.cc:385] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.657819 1178 raft_consensus.cc:740] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.658079 1178 consensus_queue.cc:260] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.658391 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:29.658576 774 raft_consensus.cc:2468] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:29.658771 1178 ts_tablet_manager.cc:1434] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:29.658120 1177 raft_consensus.cc:359] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.659183 1177 raft_consensus.cc:385] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.659289 1177 raft_consensus.cc:740] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.659685 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1071) took 2588 ms. Trace:
I20251029 08:52:29.659906 1177 consensus_queue.cc:260] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.659842 1179 rpcz_store.cc:276] 1029 08:52:27.071633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.073881 (+ 2248us) service_pool.cc:225] Handling call
1029 08:52:29.572835 (+2498954us) ts_tablet_manager.cc:1151] Deleting tablet af913080bfe44a86a76704ffa32cd9e1
1029 08:52:29.572845 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.620572 (+ 47727us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.623740 (+ 3168us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.659661 (+ 35921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498792}
I20251029 08:52:29.660624 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1079) took 2397 ms. Trace:
I20251029 08:52:29.660797 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet af913080bfe44a86a76704ffa32cd9e1 (table default.Table_7fd53f3b59d14ad39715bc62e513ab79 [id=88bde60591764b37b719bdeea15ef0cd]) successfully deleted
I20251029 08:52:29.660776 1179 rpcz_store.cc:276] 1029 08:52:27.263512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.263556 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:29.660492 (+2396936us) ts_tablet_manager.cc:1151] Deleting tablet bd357a723a1d4bc1aa4b480c8a29ef03
1029 08:52:29.660501 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.660606 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2396584}
I20251029 08:52:29.660939 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:29.661298 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bd357a723a1d4bc1aa4b480c8a29ef03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd357a723a1d4bc1aa4b480c8a29ef03 already in progress: creating tablet
I20251029 08:52:29.661648 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1080) took 2398 ms. Trace:
I20251029 08:52:29.661727 1179 rpcz_store.cc:276] 1029 08:52:27.263612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.267964 (+ 4352us) service_pool.cc:225] Handling call
1029 08:52:29.661540 (+2393576us) ts_tablet_manager.cc:1151] Deleting tablet 2fad7549758649229ad771de926aaf15
1029 08:52:29.661551 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.661632 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393372}
I20251029 08:52:29.662045 1179 tablet_replica.cc:333] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:29.662171 998 leader_election.cc:304] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:29.662273 1179 raft_consensus.cc:2243] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.662405 1179 raft_consensus.cc:2272] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.662607 2797 raft_consensus.cc:2804] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:29.662693 2797 raft_consensus.cc:493] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:29.662776 2797 raft_consensus.cc:3060] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:29.662781 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2fad7549758649229ad771de926aaf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fad7549758649229ad771de926aaf15 already in progress: creating tablet
W20251029 08:52:29.662794 996 leader_election.cc:343] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.663291 1179 ts_tablet_manager.cc:1916] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.664680 1178 tablet_bootstrap.cc:492] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.665484 2797 raft_consensus.cc:515] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.666069 2797 leader_election.cc:290] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.666637 1178 tablet_bootstrap.cc:654] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.670653 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:29.670840 774 raft_consensus.cc:3060] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.671113 1177 ts_tablet_manager.cc:1434] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 08:52:29.671682 1177 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.673508 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_83f92101d57a49da80bdc2552b39a1ba [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 1 got 0
W20251029 08:52:29.673887 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 739f3ef68dc141af90954bf0da4c5d2e consensus state has no leader
I20251029 08:52:29.673849 886 tablet_service.cc:1505] Processing CreateTablet for tablet e63d72eb9ecc4a01beb130760273cedb (DEFAULT_TABLE table=default.Table_afe3884a169642a99f2dbbeb57bb8d9d [id=5c2d6d50601c4645bb3816cbc11e01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.674141 597 master_service.cc:954] ReplaceTablet: received request to replace tablet 8017e8fe5f634ef0bf7f8be7c8aea033 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.674218 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63d72eb9ecc4a01beb130760273cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.675500 588 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_afa011ccbb98416495ed91cfb02d50fb [id=32fdba980db34781a7f359080cb1e56c] alter complete (version 3)
W20251029 08:52:29.676260 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce4a3503830c481797133028f5f123d9 consensus state has no leader
I20251029 08:52:29.676326 1177 tablet_bootstrap.cc:654] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.676589 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6d00dde368eb4b14bb49537997d2a49f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.677038 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d00dde368eb4b14bb49537997d2a49f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.677924 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dd62833c9d548adabbbf71856a4b181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.677989 774 raft_consensus.cc:2468] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:29.678126 904 tablet_service.cc:1552] Processing DeleteTablet for tablet aa411391c63146e7acb8b4c6edbe12a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.678093 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_5cf067a75b3d44d5833cbac537f853eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.678284 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65b3b1c8d14743aca1465d95ace95f with delete_type TABLET_DATA_DELETED (replaced by tablet a186df0db24c4d22abee7ec57bcb199b) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:29.678522 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cf067a75b3d44d5833cbac537f853eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.683624 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c with delete_type TABLET_DATA_DELETED (replaced by tablet d51192ae278d46d4b169e2ffd76e9d7c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.685734 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcaa3fa779824b38887ae001623ba67b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:29.686161 830 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40435
W20251029 08:52:29.687553 996 leader_election.cc:343] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.688059 998 leader_election.cc:304] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.688349 2797 raft_consensus.cc:2804] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.688572 2797 raft_consensus.cc:697] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:29.688902 2797 consensus_queue.cc:237] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.695605 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet b79ec90d93ee48abaaed3bc3cd41bf62
I20251029 08:52:29.696261 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 297945c50119432a9f9d3e356d573879
I20251029 08:52:29.696887 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e0618293cfb14d9ab93a0f5f6fb1dd66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.697621 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5d292c29d2b34086a5976da251f9ed5e
I20251029 08:52:29.698029 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet adba5919a81348bb94569ca52fa4bded
I20251029 08:52:29.698484 590 master_service.cc:954] ReplaceTablet: received request to replace tablet 3f25fe685db94f3ba2003bc7e417277c from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.699920 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_77007e3fc52446939ef29f8189facb05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.700388 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77007e3fc52446939ef29f8189facb05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.701512 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 76dcdd90e4a54cbfb0cb62192c465c40
I20251029 08:52:29.701598 1177 tablet_bootstrap.cc:492] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:29.701699 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 297945c50119432a9f9d3e356d573879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.701740 1177 ts_tablet_manager.cc:1403] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251029 08:52:29.702138 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bb0e63a0c67b4931bc2282a6d83b4db1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.702553 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb0e63a0c67b4931bc2282a6d83b4db1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.702785 591 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:29.702728 1177 raft_consensus.cc:359] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.702888 1177 raft_consensus.cc:385] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.702939 1177 raft_consensus.cc:740] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.703038 904 tablet_service.cc:1552] Processing DeleteTablet for tablet adba5919a81348bb94569ca52fa4bded with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.703217 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d292c29d2b34086a5976da251f9ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.703647 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.704231 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8017e8fe5f634ef0bf7f8be7c8aea033
I20251029 08:52:29.704650 597 catalog_manager.cc:6719] ReplaceTablet: tablet 8017e8fe5f634ef0bf7f8be7c8aea033 deleted and replaced by tablet c21cfbfbde5541a8976aa9a9b50e8867
I20251029 08:52:29.704905 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bbe82478d4d64216b1b2fbc3e9c3aade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.705190 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:29.705291 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbe82478d4d64216b1b2fbc3e9c3aade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.705368 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 297945c50119432a9f9d3e356d573879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.705520 745 tablet_service.cc:1552] Processing DeleteTablet for tablet adba5919a81348bb94569ca52fa4bded with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.705633 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d292c29d2b34086a5976da251f9ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.705787 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.705943 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 with delete_type TABLET_DATA_DELETED (replaced by tablet c21cfbfbde5541a8976aa9a9b50e8867) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.706027 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 with delete_type TABLET_DATA_DELETED (replaced by tablet c21cfbfbde5541a8976aa9a9b50e8867) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.706074 1176 ts_tablet_manager.cc:1929] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.706179 1176 log.cc:1199] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0c2072c16fce43278f1e32b9672187e7
W20251029 08:52:29.706415 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0618293cfb14d9ab93a0f5f6fb1dd66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.706692 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 297945c50119432a9f9d3e356d573879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.706890 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d292c29d2b34086a5976da251f9ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.706923 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet adba5919a81348bb94569ca52fa4bded with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.706966 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 76dcdd90e4a54cbfb0cb62192c465c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.706966 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.708441 1183 tablet_bootstrap.cc:492] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.708451 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_f87314ab36174c9184d9a2cb9e00b51f [id=b328433950f0429bb15733b81ea887df] alter complete (version 1)
I20251029 08:52:29.708570 1183 ts_tablet_manager.cc:1403] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.005s
I20251029 08:52:29.708947 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_20e164b8395a4054a8c704d344a20a57" } new_table_name: "default.Table_f38c5c596ef148eb8ac7ea611a4e4b3d" modify_external_catalogs: true
I20251029 08:52:29.710352 1183 raft_consensus.cc:359] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.710522 1183 raft_consensus.cc:385] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.710582 1183 raft_consensus.cc:740] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.710769 1183 consensus_queue.cc:260] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.710983 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5a446210f03f460b98bbab03e6a98948" } new_table_name: "default.Table_9537a25b5843488db582ad1a9224b1a2" modify_external_catalogs: true
I20251029 08:52:29.711249 1183 ts_tablet_manager.cc:1434] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20251029 08:52:29.711511 1183 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.712392 886 tablet_service.cc:1552] Processing DeleteTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.713136 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3f25fe685db94f3ba2003bc7e417277c
I20251029 08:52:29.713547 590 catalog_manager.cc:6719] ReplaceTablet: tablet 3f25fe685db94f3ba2003bc7e417277c deleted and replaced by tablet b3de9bd5a00e42c28c8eadc36795b286
I20251029 08:52:29.713816 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7da2a2ba63ee4190bbe7e36d1cac99bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.714236 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7da2a2ba63ee4190bbe7e36d1cac99bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.714766 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.714926 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f25fe685db94f3ba2003bc7e417277c with delete_type TABLET_DATA_DELETED (replaced by tablet b3de9bd5a00e42c28c8eadc36795b286) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.715058 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.715080 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 with delete_type TABLET_DATA_DELETED (replaced by tablet c21cfbfbde5541a8976aa9a9b50e8867) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.715209 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.715898 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 85df43bf781c421688f163331de0eb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.716089 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f25fe685db94f3ba2003bc7e417277c with delete_type TABLET_DATA_DELETED (replaced by tablet b3de9bd5a00e42c28c8eadc36795b286) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.716486 1177 consensus_queue.cc:260] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.718757 1177 ts_tablet_manager.cc:1434] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
W20251029 08:52:29.719281 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bc9ac40dd5b49578945e3e68e235c04 consensus state has no leader
W20251029 08:52:29.719677 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bfa85dfcdad46d3a7cbc7150c3b625f consensus state has no leader
I20251029 08:52:29.719980 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cbee24e26857417db3a2cb72aca81cb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.721140 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbee24e26857417db3a2cb72aca81cb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.722298 589 catalog_manager.cc:5649] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:29.722695 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_a5ebbce3f0ef404699dae9a0c1c670c3 [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 2 got 0
I20251029 08:52:29.722934 1177 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:29.723104 589 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:29.724736 2554 raft_consensus.cc:493] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.724858 2554 raft_consensus.cc:515] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.725117 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f25fe685db94f3ba2003bc7e417277c with delete_type TABLET_DATA_DELETED (replaced by tablet b3de9bd5a00e42c28c8eadc36795b286) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.725437 2554 leader_election.cc:290] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.726246 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5ac1af8b78a4d94b0e3da14ebfffd92" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.726442 1067 raft_consensus.cc:2468] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:29.726562 861 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:29.726979 861 leader_election.cc:304] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:29.727690 2554 raft_consensus.cc:2804] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:29.727766 2554 raft_consensus.cc:493] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:29.727843 2554 raft_consensus.cc:3060] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.728952 1176 ts_tablet_manager.cc:1950] T 0c2072c16fce43278f1e32b9672187e7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.729809 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1089) took 2416 ms. Trace:
I20251029 08:52:29.729893 1176 rpcz_store.cc:276] 1029 08:52:27.313115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.313607 (+ 492us) service_pool.cc:225] Handling call
1029 08:52:29.636735 (+2323128us) ts_tablet_manager.cc:1151] Deleting tablet 0c2072c16fce43278f1e32b9672187e7
1029 08:52:29.636747 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.679458 (+ 42711us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.705967 (+ 26509us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.729789 (+ 23822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322748}
I20251029 08:52:29.730289 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0c2072c16fce43278f1e32b9672187e7 (table default.Table_bceccce7081b43f6ad2849ed269dfe73 [id=468fee62b8ce4f0dad6500dcfe3c70c6]) successfully deleted
I20251029 08:52:29.730720 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1091) took 2415 ms. Trace:
I20251029 08:52:29.730798 1176 rpcz_store.cc:276] 1029 08:52:27.315226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.315273 (+ 47us) service_pool.cc:225] Handling call
1029 08:52:29.730588 (+2415315us) ts_tablet_manager.cc:1151] Deleting tablet f744556dccf348fd8b92071f224730aa
1029 08:52:29.730598 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.730702 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415144}
W20251029 08:52:29.731165 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f744556dccf348fd8b92071f224730aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f744556dccf348fd8b92071f224730aa already in progress: creating tablet
I20251029 08:52:29.730840 2554 raft_consensus.cc:515] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.731801 2554 leader_election.cc:290] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.733381 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5ac1af8b78a4d94b0e3da14ebfffd92" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:29.733592 1067 raft_consensus.cc:3060] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.734771 1183 tablet_bootstrap.cc:654] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.736739 1067 raft_consensus.cc:2468] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:29.737107 861 leader_election.cc:304] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
W20251029 08:52:29.737558 861 leader_election.cc:343] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:29.737885 2554 raft_consensus.cc:2804] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.738104 2554 raft_consensus.cc:697] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:29.738372 2554 consensus_queue.cc:237] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.741778 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1092) took 2426 ms. Trace:
I20251029 08:52:29.741868 1176 rpcz_store.cc:276] 1029 08:52:27.315655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.315700 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:29.731473 (+2415773us) ts_tablet_manager.cc:1151] Deleting tablet cea6d85081f246788ba29bb4a9c9a4e8
1029 08:52:29.731483 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.741757 (+ 10274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415630}
W20251029 08:52:29.742274 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cea6d85081f246788ba29bb4a9c9a4e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea6d85081f246788ba29bb4a9c9a4e8 already in progress: creating tablet
I20251029 08:52:29.744801 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1093) took 2428 ms. Trace:
I20251029 08:52:29.746683 764 tablet_service.cc:1505] Processing CreateTablet for tablet e63d72eb9ecc4a01beb130760273cedb (DEFAULT_TABLE table=default.Table_afe3884a169642a99f2dbbeb57bb8d9d [id=5c2d6d50601c4645bb3816cbc11e01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.747084 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63d72eb9ecc4a01beb130760273cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.744931 1176 rpcz_store.cc:276] 1029 08:52:27.316007 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.316074 (+ 67us) service_pool.cc:225] Handling call
1029 08:52:29.744583 (+2428509us) ts_tablet_manager.cc:1151] Deleting tablet 38032950744b4426927157fad0962dfb
1029 08:52:29.744595 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.744759 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428334}
I20251029 08:52:29.748942 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1095) took 2428 ms. Trace:
I20251029 08:52:29.749338 1176 rpcz_store.cc:276] 1029 08:52:27.320268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.320320 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:29.748413 (+2428093us) ts_tablet_manager.cc:1151] Deleting tablet 46bcc468ce964610a857bc2b11b5b3e6
1029 08:52:29.748423 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.748832 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427916}
W20251029 08:52:29.751004 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 38032950744b4426927157fad0962dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38032950744b4426927157fad0962dfb already in progress: creating tablet
W20251029 08:52:29.751214 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 46bcc468ce964610a857bc2b11b5b3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bcc468ce964610a857bc2b11b5b3e6 already in progress: creating tablet
I20251029 08:52:29.752465 1177 tablet_bootstrap.cc:654] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.753448 1176 tablet_replica.cc:333] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.753854 2824 raft_consensus.cc:493] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8395ff5ec92c4bbda9bc6641a9fbbb6b)
I20251029 08:52:29.754110 2824 raft_consensus.cc:515] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } }
I20251029 08:52:29.754689 2763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395ff5ec92c4bbda9bc6641a9fbbb6b. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:29.754917 2763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 08:52:29.757906 591 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_e006f70bed4248f890eb94f10f2d90f8 [id=01723803a27d472b90b05ba626327d7c] alter complete (version 2)
I20251029 08:52:29.759830 2321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34668 (request call id 0) took 7237 ms. Trace:
I20251029 08:52:29.765571 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 6689cf0804f946c18ebc555a191549d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.765831 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ab56045bd814d59831a98618c759af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.766010 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 254e62d88c3242a0a3ac7470a8005d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.766263 886 tablet_service.cc:1552] Processing DeleteTablet for tablet d5ac1af8b78a4d94b0e3da14ebfffd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.770804 2824 leader_election.cc:290] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85c8cdf28bda444d851fb50588400173 (127.0.104.61:38825), 8395ff5ec92c4bbda9bc6641a9fbbb6b (127.0.104.60:42135)
I20251029 08:52:29.772622 591 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 442c37e1ee724b6b96f679f05640baf9 (table default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb]). Expected version 1 got 0
I20251029 08:52:29.776856 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_93f45fc099dd4110a562e1fc85e8045c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.777256 596 master_service.cc:954] ReplaceTablet: received request to replace tablet 919e5494b84343639a2b9b81b05f1c2c from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:29.777513 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93f45fc099dd4110a562e1fc85e8045c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.777902 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_36bb0544db8d43f88d006f794682fab7" } new_table_name: "default.Table_17212839de0141b18ba288225b77ef1a" modify_external_catalogs: true
I20251029 08:52:29.779121 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3b11dfccad424815b9e7803c8c2dd90b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.779609 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b11dfccad424815b9e7803c8c2dd90b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.781919 1046 tablet_service.cc:1505] Processing CreateTablet for tablet e63d72eb9ecc4a01beb130760273cedb (DEFAULT_TABLE table=default.Table_afe3884a169642a99f2dbbeb57bb8d9d [id=5c2d6d50601c4645bb3816cbc11e01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.782354 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63d72eb9ecc4a01beb130760273cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.787732 2554 raft_consensus.cc:493] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.787870 2554 raft_consensus.cc:515] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.791772 589 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225] alter complete (version 1)
I20251029 08:52:29.792794 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.791954 2554 leader_election.cc:290] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.797940 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet e0fa089fce564af4b03aa323cec6cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.798745 1068 raft_consensus.cc:1275] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Refusing update from remote peer 7a8d80098fcb40ada903c3b2bacc2ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 08:52:29.798765 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet 391977048e5f4f0b8f62e4d5e3d2946b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.799131 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.799343 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.799746 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:29.803354 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31c7b7fe77624dcf9b04a38948d3919c has no consensus state
I20251029 08:52:29.792744 757 tablet_service.cc:1552] Processing DeleteTablet for tablet b1aaf233eba2461f9182e094a0115a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.792754 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fb69271e14fe3a8d0b29d0db0dc8e" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:29.793277 749 tablet_service.cc:1552] Processing DeleteTablet for tablet d306be1d51674c2abc506d248e05a9c7 with delete_type TABLET_DATA_DELETED (replaced by tablet a08630cbc401467a9dde1ce1e198cef7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.811852 1176 raft_consensus.cc:2243] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.812084 1176 raft_consensus.cc:2272] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.812181 861 leader_election.cc:304] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:29.812481 2554 raft_consensus.cc:2749] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.793887 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.794401 758 tablet_service.cc:1552] Processing DeleteTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.795070 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4200db822deb4ace936f561c230ac1fb with delete_type TABLET_DATA_DELETED (replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.812644 1176 ts_tablet_manager.cc:1916] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:29.799221 718 consensus_peers.cc:597] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:29.799345 2786 consensus_queue.cc:1048] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:29.818899 2786 raft_consensus.cc:493] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.819053 2786 raft_consensus.cc:515] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.819679 2786 leader_election.cc:290] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:29.820200 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.820266 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:29.820825 718 leader_election.cc:343] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:29.821316 720 leader_election.cc:343] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.821416 720 leader_election.cc:304] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.821632 2786 raft_consensus.cc:2749] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.823611 1178 tablet_bootstrap.cc:492] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:29.823918 1178 ts_tablet_manager.cc:1403] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.159s user 0.003s sys 0.001s
I20251029 08:52:29.824827 1178 raft_consensus.cc:359] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.727993 1974 ts_tablet_manager.cc:1929] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:29.828749 1974 log.cc:1199] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/627bc8ff99394fa2abcca15d62e5ea10
I20251029 08:52:29.828784 2755 tablet.cc:1722] T ce4a3503830c481797133028f5f123d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:29.829200 1974 ts_tablet_manager.cc:1950] T 627bc8ff99394fa2abcca15d62e5ea10 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:29.829506 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1033) took 4158 ms. Trace:
I20251029 08:52:29.829578 1974 rpcz_store.cc:276] 1029 08:52:25.671365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.671446 (+ 81us) service_pool.cc:225] Handling call
1029 08:52:29.614859 (+3943413us) ts_tablet_manager.cc:1151] Deleting tablet 627bc8ff99394fa2abcca15d62e5ea10
1029 08:52:29.614869 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.619364 (+ 4495us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.727893 (+108529us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.829490 (+101597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3943235}
I20251029 08:52:29.830046 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 627bc8ff99394fa2abcca15d62e5ea10 (table default.Table_538d255b7c7e4f9fbe5f5ee8236746b7 [id=57ea37ee03ad41839bcda05f5f3fbf0a]) successfully deleted
I20251029 08:52:29.830384 1974 tablet_replica.cc:333] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:29.830598 1974 raft_consensus.cc:2243] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.830747 1974 raft_consensus.cc:2272] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:29.831414 1974 ts_tablet_manager.cc:1916] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.831519 886 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a0066c04948c6af4f77564e66832a (DEFAULT_TABLE table=default.Table_28cea2a6590a436e98fce31e8bf4eff3 [id=768b983d716e47a799e6379f4bc9fca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.831869 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a0066c04948c6af4f77564e66832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.837203 596 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_a5ebbce3f0ef404699dae9a0c1c670c3 [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 2 got 0
I20251029 08:52:29.826503 2756 tablet.cc:1722] T ce4a3503830c481797133028f5f123d9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:29.828289 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a0066c04948c6af4f77564e66832a (DEFAULT_TABLE table=default.Table_28cea2a6590a436e98fce31e8bf4eff3 [id=768b983d716e47a799e6379f4bc9fca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.846171 1178 raft_consensus.cc:385] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.846362 1178 raft_consensus.cc:740] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.842088 2780 raft_consensus.cc:493] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.846588 2780 raft_consensus.cc:515] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.846628 1178 consensus_queue.cc:260] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.847275 2780 leader_election.cc:290] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:29.847337 1178 ts_tablet_manager.cc:1434] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251029 08:52:29.848177 596 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_a5ebbce3f0ef404699dae9a0c1c670c3 [id=e1da82f219d140ad8ee2c8d2786bd40e] alter complete (version 2)
I20251029 08:52:29.848850 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d306be1d51674c2abc506d248e05a9c7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.849591 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a0066c04948c6af4f77564e66832a (DEFAULT_TABLE table=default.Table_28cea2a6590a436e98fce31e8bf4eff3 [id=768b983d716e47a799e6379f4bc9fca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.850636 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d306be1d51674c2abc506d248e05a9c7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:29.851528 718 leader_election.cc:343] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.851796 886 tablet_service.cc:1505] Processing CreateTablet for tablet cbded90ed7db437db47cfe6deba3fe3f (DEFAULT_TABLE table=default.Table_a94e89ac464f4105987c9db0d6a6f390 [id=116cb52d022747ac83ea284dd43a21a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.851907 758 tablet_service.cc:1505] Processing CreateTablet for tablet cbded90ed7db437db47cfe6deba3fe3f (DEFAULT_TABLE table=default.Table_a94e89ac464f4105987c9db0d6a6f390 [id=116cb52d022747ac83ea284dd43a21a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.852206 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbded90ed7db437db47cfe6deba3fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.856077 2797 raft_consensus.cc:493] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.856204 2797 raft_consensus.cc:515] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.856830 2797 leader_election.cc:290] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.857191 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:29.857187 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.857344 773 raft_consensus.cc:2468] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:29.857749 998 leader_election.cc:304] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:29.857923 996 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.857990 2707 raft_consensus.cc:2804] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:29.858044 2707 raft_consensus.cc:493] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251029 08:52:29.858035 720 leader_election.cc:343] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.858100 2707 raft_consensus.cc:3060] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.858139 720 leader_election.cc:304] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.858604 2780 raft_consensus.cc:493] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.858731 2780 raft_consensus.cc:515] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.859229 2780 leader_election.cc:290] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:29.859339 2780 raft_consensus.cc:2749] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:29.859864 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:29.860441 718 leader_election.cc:343] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.860718 2707 raft_consensus.cc:515] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.860936 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:29.861239 2707 leader_election.cc:290] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:29.861276 720 leader_election.cc:343] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.861362 720 leader_election.cc:304] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.861572 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:29.861693 2786 raft_consensus.cc:2749] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:29.861984 996 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.862389 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7046bb1fcdd34d308aecddfdf0752911" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:29.862540 773 raft_consensus.cc:3060] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:29.862643 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a0066c04948c6af4f77564e66832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.865355 773 raft_consensus.cc:2468] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:29.865319 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef65af05225438db146adb4dbca4bff (DEFAULT_TABLE table=default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3 [id=0af0423774cc4213b01562ed89794b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.866178 901 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef65af05225438db146adb4dbca4bff (DEFAULT_TABLE table=default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3 [id=0af0423774cc4213b01562ed89794b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.866554 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef65af05225438db146adb4dbca4bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.866855 1044 tablet_service.cc:1505] Processing CreateTablet for tablet d60d5c8081894230b77372e3af39eed2 (DEFAULT_TABLE table=default.Table_63f64af28ee546ca9bd6d0c3c75cf732 [id=72f0f8d5c4f74e60af3f43568ab5ba28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.868266 1050 tablet_service.cc:1505] Processing CreateTablet for tablet cbded90ed7db437db47cfe6deba3fe3f (DEFAULT_TABLE table=default.Table_a94e89ac464f4105987c9db0d6a6f390 [id=116cb52d022747ac83ea284dd43a21a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.868657 1047 tablet_service.cc:1505] Processing CreateTablet for tablet ac09bb4502e440368633bbc90e9ad2cf (DEFAULT_TABLE table=default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5 [id=16acd998dd44401d96398bbdb8be8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.869481 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbded90ed7db437db47cfe6deba3fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.869529 1178 tablet_bootstrap.cc:492] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:29.869660 745 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef65af05225438db146adb4dbca4bff (DEFAULT_TABLE table=default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3 [id=0af0423774cc4213b01562ed89794b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.870039 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef65af05225438db146adb4dbca4bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.870371 764 tablet_service.cc:1505] Processing CreateTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.870684 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5525b69c8a4450b6d6571d39abb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.872251 749 tablet_service.cc:1505] Processing CreateTablet for tablet d60d5c8081894230b77372e3af39eed2 (DEFAULT_TABLE table=default.Table_63f64af28ee546ca9bd6d0c3c75cf732 [id=72f0f8d5c4f74e60af3f43568ab5ba28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.872574 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d5c8081894230b77372e3af39eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.876219 757 tablet_service.cc:1505] Processing CreateTablet for tablet ac09bb4502e440368633bbc90e9ad2cf (DEFAULT_TABLE table=default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5 [id=16acd998dd44401d96398bbdb8be8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.876513 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac09bb4502e440368633bbc90e9ad2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.877650 1178 tablet_bootstrap.cc:654] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.882525 998 leader_election.cc:304] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:29.882830 2797 raft_consensus.cc:2804] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:29.882954 2797 raft_consensus.cc:697] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:29.883221 2797 consensus_queue.cc:237] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.886013 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_073a0e8a44f04947b524dac0cdb01fb1" } new_table_name: "default.Table_240517c7375143668c05aa5314f370ef" modify_external_catalogs: true
W20251029 08:52:29.892679 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd393400e8b2473482ab78008863c48b consensus state has no leader
I20251029 08:52:29.898797 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef65af05225438db146adb4dbca4bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.899247 2833 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 08:52:29.900233 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d5c8081894230b77372e3af39eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.901945 1183 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.902071 1183 ts_tablet_manager.cc:1403] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.002s
I20251029 08:52:29.902801 1183 raft_consensus.cc:359] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.902948 1183 raft_consensus.cc:385] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.902989 1183 raft_consensus.cc:740] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.903187 1183 consensus_queue.cc:260] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.904238 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbded90ed7db437db47cfe6deba3fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.907193 1183 ts_tablet_manager.cc:1434] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:29.907434 1183 tablet_bootstrap.cc:492] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.908164 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac09bb4502e440368633bbc90e9ad2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.909965 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ccddc094f01348a5b25ebd260db6d8cd" } modify_external_catalogs: true
I20251029 08:52:29.910122 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ccddc094f01348a5b25ebd260db6d8cd" } modify_external_catalogs: true
I20251029 08:52:29.911334 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_af67eaef7a72468c86b5c55ccb6d2626" } modify_external_catalogs: true
I20251029 08:52:29.911489 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_af67eaef7a72468c86b5c55ccb6d2626" } modify_external_catalogs: true
I20251029 08:52:29.912653 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d2361c16d6164246a7b81385f8fcf1c6" } modify_external_catalogs: true
I20251029 08:52:29.912787 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d2361c16d6164246a7b81385f8fcf1c6" } modify_external_catalogs: true
I20251029 08:52:29.916953 1183 tablet_bootstrap.cc:654] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.922247 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b2a8ce7300d4dfda4127e313d914f22" } new_table_name: "default.Table_07159bd6d35d4ba3b84c2c8c27ac779b" modify_external_catalogs: true
I20251029 08:52:29.923570 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e3fab7c3fd2644f88ad0d6f2395cb4b3" } new_table_name: "default.Table_e84c903c0e004754ae80e72bfa495a8c" modify_external_catalogs: true
I20251029 08:52:29.927538 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ecb4a70de9424d629eaa74f4be4c6549" } modify_external_catalogs: true
I20251029 08:52:29.927680 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ecb4a70de9424d629eaa74f4be4c6549" } modify_external_catalogs: true
I20251029 08:52:29.930979 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ea40ed38c0ba4d2f9045188af226ced5
I20251029 08:52:29.931320 2707 raft_consensus.cc:493] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.931445 2707 raft_consensus.cc:515] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.932082 2707 leader_election.cc:290] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:29.932142 757 tablet_service.cc:1552] Processing DeleteTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.933257 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 307262da8888461ebddaaf33a31be8b8
I20251029 08:52:29.934057 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 307262da8888461ebddaaf33a31be8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.934245 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 307262da8888461ebddaaf33a31be8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.934345 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4675f0aec6a14807afd3094780d60c72
W20251029 08:52:29.935355 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ef65af05225438db146adb4dbca4bff consensus state has no leader
W20251029 08:52:29.935827 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad8a88123f8542eea86d8656bfef97df consensus state has no leader
I20251029 08:52:29.936362 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 4675f0aec6a14807afd3094780d60c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.936475 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 4675f0aec6a14807afd3094780d60c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:29.938794 998 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 65b20d53d6214bdc81ca205dbf922d7b
I20251029 08:52:29.946158 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a0066c04948c6af4f77564e66832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:29.951866 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4524a248859448c5b789c7ae82435705
I20251029 08:52:29.952800 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 4524a248859448c5b789c7ae82435705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.953835 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.954010 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 4524a248859448c5b789c7ae82435705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.954260 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_073dbf71123c4416bb365911b2d93ad6" } modify_external_catalogs: true
I20251029 08:52:29.954926 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_073dbf71123c4416bb365911b2d93ad6" } modify_external_catalogs: true
I20251029 08:52:29.957239 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.962431 591 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 78ca9fc5dfcd4975b642695806d61de6
I20251029 08:52:29.963165 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 1156 } } ignore_live_leader: false dest_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" is_pre_election: true
I20251029 08:52:29.969106 1183 tablet_bootstrap.cc:492] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:29.969310 1183 ts_tablet_manager.cc:1403] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.007s
I20251029 08:52:29.970221 904 tablet_service.cc:1505] Processing CreateTablet for tablet d60d5c8081894230b77372e3af39eed2 (DEFAULT_TABLE table=default.Table_63f64af28ee546ca9bd6d0c3c75cf732 [id=72f0f8d5c4f74e60af3f43568ab5ba28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.970829 1183 raft_consensus.cc:359] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.970984 1183 raft_consensus.cc:385] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:29.971040 1183 raft_consensus.cc:740] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:29.971235 1183 consensus_queue.cc:260] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:29.971767 1183 ts_tablet_manager.cc:1434] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:29.972095 1183 tablet_bootstrap.cc:492] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:29.972507 896 tablet_service.cc:1505] Processing CreateTablet for tablet ac09bb4502e440368633bbc90e9ad2cf (DEFAULT_TABLE table=default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5 [id=16acd998dd44401d96398bbdb8be8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:29.973001 2303 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" instance_seqno: 1761727921574165) as {username='slave'} at 127.0.104.3:47163; Asking this server to re-register.
I20251029 08:52:29.973734 895 tablet_service.cc:1552] Processing DeleteTablet for tablet ea40ed38c0ba4d2f9045188af226ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:29.974160 1183 tablet_bootstrap.cc:654] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:29.974999 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b20d53d6214bdc81ca205dbf922d7b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:29.975850 1176 ts_tablet_manager.cc:1929] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:29.975955 1176 log.cc:1199] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0bd490da0b0e47fc965343c123a27b22
I20251029 08:52:29.976248 1176 ts_tablet_manager.cc:1950] T 0bd490da0b0e47fc965343c123a27b22 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:29.978674 591 master_service.cc:954] ReplaceTablet: received request to replace tablet f2c18e4c41714ba38bdd2e208a5e761e from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:29.980522 757 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:29.981041 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 1156 } } ignore_live_leader: false dest_uuid: "85c8cdf28bda444d851fb50588400173" is_pre_election: true
I20251029 08:52:29.981842 2259 leader_election.cc:304] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f718bd0f00004f20a89d028a2fa8e6b9; no voters: 8395ff5ec92c4bbda9bc6641a9fbbb6b, 85c8cdf28bda444d851fb50588400173
I20251029 08:52:29.982424 2824 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251029 08:52:29.983914 2786 raft_consensus.cc:493] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:29.984071 2786 raft_consensus.cc:515] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:29.984645 2786 leader_election.cc:290] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:29.986438 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:29.986992 720 leader_election.cc:343] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:29.988165 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_71e81aba893d4388889abc1b3d7afebb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.988740 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71e81aba893d4388889abc1b3d7afebb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.990602 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d39067d9fd47411f95a64ff74c3c4df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.991138 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d39067d9fd47411f95a64ff74c3c4df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.992378 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_48c90ff8955448b891db2bf3482a489c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.992924 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c90ff8955448b891db2bf3482a489c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.993835 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_05c08c8aca93406dadacfdc745eae1e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.994372 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05c08c8aca93406dadacfdc745eae1e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.994369 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet b1aaf233eba2461f9182e094a0115a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:29.995944 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_39ab2d051bdd4c3a8bbf20b2178eba44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:29.995967 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_697b189fcd394daa8cb0fdb8d0c87518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.996331 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39ab2d051bdd4c3a8bbf20b2178eba44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:29.996507 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_697b189fcd394daa8cb0fdb8d0c87518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.997526 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3edf34ad561b40ef83f837d36eb91335"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:29.997920 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3edf34ad561b40ef83f837d36eb91335 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:29.998714 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1090) took 2683 ms. Trace:
I20251029 08:52:29.998819 1176 rpcz_store.cc:276] 1029 08:52:27.314913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.329190 (+ 14277us) service_pool.cc:225] Handling call
1029 08:52:29.750118 (+2420928us) ts_tablet_manager.cc:1151] Deleting tablet 0bd490da0b0e47fc965343c123a27b22
1029 08:52:29.750129 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.960080 (+209951us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.975745 (+ 15665us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:29.998690 (+ 22945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420150}
I20251029 08:52:29.999264 1176 tablet_replica.cc:333] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:29.999492 1176 raft_consensus.cc:2243] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:29.999486 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0bd490da0b0e47fc965343c123a27b22 (table default.Table_a97711da23ff4ab79b7f1810e23031c6 [id=0d9f935637a8420b985538e9de4e21ca]) successfully deleted
I20251029 08:52:29.999610 1176 raft_consensus.cc:2272] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.000092 1176 ts_tablet_manager.cc:1916] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.999092 1178 tablet_bootstrap.cc:492] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:30.002573 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.003042 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5525b69c8a4450b6d6571d39abb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.006974 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_18732ceb5df84aa88ea943c4d35e6084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.007375 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18732ceb5df84aa88ea943c4d35e6084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.008567 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d101fe184fa848c19ec0ca5cc80ae445"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.008965 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d101fe184fa848c19ec0ca5cc80ae445 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.010186 2303 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" instance_seqno: 1761727920315454) as {username='slave'} at 127.0.104.1:41145; Asking this server to re-register.
I20251029 08:52:30.013286 1178 ts_tablet_manager.cc:1403] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.000s
I20251029 08:52:30.014068 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c0e6b2aa7a434d399ef4384fe8093945" } new_table_name: "default.Table_4498572fb2e34861a6de39a0b65d120f" modify_external_catalogs: true
I20251029 08:52:30.016312 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f2c18e4c41714ba38bdd2e208a5e761e
I20251029 08:52:30.016891 591 catalog_manager.cc:6719] ReplaceTablet: tablet f2c18e4c41714ba38bdd2e208a5e761e deleted and replaced by tablet 6619c659ed8c43abb71c4222490ae0d9
I20251029 08:52:30.017283 757 tablet_service.cc:1552] Processing DeleteTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e with delete_type TABLET_DATA_DELETED (replaced by tablet 6619c659ed8c43abb71c4222490ae0d9) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.017436 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ec5782dfc71a40e5870d6fc9a8e51780"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.017692 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e with delete_type TABLET_DATA_DELETED (replaced by tablet 6619c659ed8c43abb71c4222490ae0d9) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:30.017854 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec5782dfc71a40e5870d6fc9a8e51780 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.019440 757 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac8d52f6e954e96aa30f20c33d5d81d (DEFAULT_TABLE table=default.Table_36f1ed1c59f947ceaba03af7ec3bf4e0 [id=ad070022fee84954aaf518fc621e7172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.019819 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac8d52f6e954e96aa30f20c33d5d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.025051 830 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:30.028573 2303 ts_manager.cc:194] Registered new tserver with Master: 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.030156 905 tablet_service.cc:1505] Processing CreateTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.032444 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 307262da8888461ebddaaf33a31be8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.032512 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 4524a248859448c5b789c7ae82435705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.032629 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_2a5ef8abeebf415084663ef00f8d942b" } new_table_name: "default.Table_0eea57b1b6c848cb818d34551aef2c3e" modify_external_catalogs: true
I20251029 08:52:30.032464 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 4675f0aec6a14807afd3094780d60c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:29 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.033069 2554 raft_consensus.cc:493] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:30.033787 996 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.033898 996 leader_election.cc:304] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.034238 2797 raft_consensus.cc:2749] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.035977 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.037086 891 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac8d52f6e954e96aa30f20c33d5d81d (DEFAULT_TABLE table=default.Table_36f1ed1c59f947ceaba03af7ec3bf4e0 [id=ad070022fee84954aaf518fc621e7172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.037389 897 tablet_service.cc:1552] Processing DeleteTablet for tablet f2c18e4c41714ba38bdd2e208a5e761e with delete_type TABLET_DATA_DELETED (replaced by tablet 6619c659ed8c43abb71c4222490ae0d9) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:30.038278 718 leader_election.cc:343] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.038376 718 leader_election.cc:304] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.038697 2786 raft_consensus.cc:2749] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.040783 1179 ts_tablet_manager.cc:1929] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.040895 1179 log.cc:1199] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/657cdaad97e24611a997e26a6293ea5e
I20251029 08:52:30.041193 1179 ts_tablet_manager.cc:1950] T 657cdaad97e24611a997e26a6293ea5e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.042047 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1081) took 2773 ms. Trace:
I20251029 08:52:30.042135 1179 rpcz_store.cc:276] 1029 08:52:27.268373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.268572 (+ 199us) service_pool.cc:225] Handling call
1029 08:52:29.662005 (+2393433us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:29.662013 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:29.883117 (+221104us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.040683 (+157566us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.042028 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393276}
W20251029 08:52:30.050366 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9d6d7b3e06144cbb1cabec1a9360340 consensus state has no leader
I20251029 08:52:30.033193 2554 raft_consensus.cc:515] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.052363 921 raft_consensus.cc:3060] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.052071 1117 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:30.066955 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 657cdaad97e24611a997e26a6293ea5e (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281]) successfully deleted
I20251029 08:52:30.067248 1179 tablet_replica.cc:333] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.067566 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d5c8081894230b77372e3af39eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.068953 1179 raft_consensus.cc:2243] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.069506 2554 leader_election.cc:290] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.070628 1179 raft_consensus.cc:2272] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.072126 2303 ts_manager.cc:194] Registered new tserver with Master: 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:30.074393 1176 ts_tablet_manager.cc:1929] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.074543 1176 log.cc:1199] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/f5b52314139945dd9c2e6ec899bf06dc
I20251029 08:52:30.074887 1183 tablet_bootstrap.cc:492] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.074919 1176 ts_tablet_manager.cc:1950] T f5b52314139945dd9c2e6ec899bf06dc P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.075008 1183 ts_tablet_manager.cc:1403] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.008s
I20251029 08:52:30.074451 1179 ts_tablet_manager.cc:1916] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.075709 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1098) took 2655 ms. Trace:
I20251029 08:52:30.075670 1183 raft_consensus.cc:359] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.075822 1183 raft_consensus.cc:385] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.075873 1183 raft_consensus.cc:740] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.075819 1176 rpcz_store.cc:276] 1029 08:52:27.420448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.422997 (+ 2549us) service_pool.cc:225] Handling call
1029 08:52:29.999203 (+2576206us) ts_tablet_manager.cc:1151] Deleting tablet f5b52314139945dd9c2e6ec899bf06dc
1029 08:52:29.999217 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.043751 (+ 44534us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.055533 (+ 11782us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.075688 (+ 20155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574245}
I20251029 08:52:30.076267 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1099) took 2579 ms. Trace:
I20251029 08:52:30.076308 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet f5b52314139945dd9c2e6ec899bf06dc (table default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c [id=0a703b25b63d4578b7eff94008a90dea]) successfully deleted
I20251029 08:52:30.076346 1176 rpcz_store.cc:276] 1029 08:52:27.496720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.500619 (+ 3899us) service_pool.cc:225] Handling call
1029 08:52:30.076143 (+2575524us) ts_tablet_manager.cc:1151] Deleting tablet 960f17ab6bed4ef5ad382421d84bf8d7
1029 08:52:30.076156 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.076252 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575315}
I20251029 08:52:30.076603 1176 tablet_replica.cc:333] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
W20251029 08:52:30.076957 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 960f17ab6bed4ef5ad382421d84bf8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960f17ab6bed4ef5ad382421d84bf8d7 already in progress: creating tablet
I20251029 08:52:30.078930 1183 consensus_queue.cc:260] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.079422 1183 ts_tablet_manager.cc:1434] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:30.083036 1178 raft_consensus.cc:359] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.083230 1178 raft_consensus.cc:385] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.083287 1178 raft_consensus.cc:740] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.086099 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac8d52f6e954e96aa30f20c33d5d81d (DEFAULT_TABLE table=default.Table_36f1ed1c59f947ceaba03af7ec3bf4e0 [id=ad070022fee84954aaf518fc621e7172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.088114 1178 consensus_queue.cc:260] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.089304 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac8d52f6e954e96aa30f20c33d5d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.089838 1178 ts_tablet_manager.cc:1434] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20251029 08:52:30.090409 1176 raft_consensus.cc:2243] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:30.090847 1176 raft_consensus.cc:2272] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.091815 1178 tablet_bootstrap.cc:492] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:30.092330 764 tablet_service.cc:1505] Processing CreateTablet for tablet 213fc61402784f19a5287fdb57719616 (DEFAULT_TABLE table=default.Table_b17b82921a624ff5a4c04ddaa4462781 [id=32886fb555f5419b937be2cc966e82fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.095825 1176 ts_tablet_manager.cc:1916] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.096071 590 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 15dae38bcced496e90ca8eec280a8317 (table default.Table_1c14de238c434c4084a2b70bdf36d9a1 [id=22cd3b05e2c74671a965cd770dcee5a9]). Expected version 1 got 0
I20251029 08:52:30.097292 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213fc61402784f19a5287fdb57719616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.101184 897 tablet_service.cc:1505] Processing CreateTablet for tablet 213fc61402784f19a5287fdb57719616 (DEFAULT_TABLE table=default.Table_b17b82921a624ff5a4c04ddaa4462781 [id=32886fb555f5419b937be2cc966e82fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.104794 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_cdae86f1af1840569f67820138d741ea" } new_table_name: "default.Table_4949f864d88f46d6b3e1ef34bbfc5418" modify_external_catalogs: true
I20251029 08:52:30.109473 2797 raft_consensus.cc:493] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.109620 2797 raft_consensus.cc:515] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.110176 2797 leader_election.cc:290] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:30.117633 998 leader_election.cc:343] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 3cd2e9a95b6c40be9e17397a313d249b
I20251029 08:52:30.118577 2755 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 08:52:30.119601 2756 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
W20251029 08:52:30.123521 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4cb38b4023242fa8e85356dd4cccd70 consensus state has no leader
I20251029 08:52:30.125600 596 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 15dae38bcced496e90ca8eec280a8317 (table default.Table_1c14de238c434c4084a2b70bdf36d9a1 [id=22cd3b05e2c74671a965cd770dcee5a9]). Expected version 1 got 0
I20251029 08:52:30.131873 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf9e36c47be4509a998be8e7138475b with delete_type TABLET_DATA_DELETED (replaced by tablet f284bc30fd1144d69e0e11ea648c5ed7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.132493 1179 ts_tablet_manager.cc:1929] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.132635 1179 log.cc:1199] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:30.133037 1179 ts_tablet_manager.cc:1950] T 19fdc381a0d949b0b93cef399daa3021 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.132721 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4c77960427da4195879204c4bd7833a2" } modify_external_catalogs: true
I20251029 08:52:30.133108 1176 ts_tablet_manager.cc:1929] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.133219 1176 log.cc:1199] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:30.133221 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4c77960427da4195879204c4bd7833a2" } modify_external_catalogs: true
I20251029 08:52:30.133571 1176 ts_tablet_manager.cc:1950] T 913fb69271e14fe3a8d0b29d0db0dc8e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.133843 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1082) took 2864 ms. Trace:
I20251029 08:52:30.133913 1179 rpcz_store.cc:276] 1029 08:52:27.268913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.269093 (+ 180us) service_pool.cc:225] Handling call
1029 08:52:30.067002 (+2797909us) ts_tablet_manager.cc:1151] Deleting tablet 19fdc381a0d949b0b93cef399daa3021
1029 08:52:30.067184 (+ 182us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.088862 (+ 21678us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.117467 (+ 28605us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.133825 (+ 16358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2797668}
I20251029 08:52:30.134286 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 19fdc381a0d949b0b93cef399daa3021 (table default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]) successfully deleted
I20251029 08:52:30.134732 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ac500003d4a84f739ff1c0ac8d40e93c" } modify_external_catalogs: true
I20251029 08:52:30.134897 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ac500003d4a84f739ff1c0ac8d40e93c" } modify_external_catalogs: true
I20251029 08:52:30.137588 597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d868be96231b420db67789c6ba1d5d2a" } modify_external_catalogs: true
I20251029 08:52:30.137637 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_dd61164773f54438b96f450cc899ab1a" } new_table_name: "default.Table_b944404172d143fa9a97aee02cc35657" modify_external_catalogs: true
I20251029 08:52:30.137712 597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_d868be96231b420db67789c6ba1d5d2a" } modify_external_catalogs: true
I20251029 08:52:30.138775 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7831bec582164564bcbadb55b6110645" } new_table_name: "default.Table_1c0fa4716ab14f70b36d4f0ffe00fb93" modify_external_catalogs: true
I20251029 08:52:30.141396 1179 tablet_replica.cc:333] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.141592 1179 raft_consensus.cc:2243] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.141906 1179 raft_consensus.cc:2272] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.142416 1179 ts_tablet_manager.cc:1916] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:29.759959 2321 rpcz_store.cc:276] 1029 08:52:22.522105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:22.522393 (+ 288us) service_pool.cc:225] Handling call
1029 08:52:28.845588 (+6323195us) raft_consensus.cc:1471] Updating replica for 281 ops
1029 08:52:29.144658 (+299070us) raft_consensus.cc:1534] Early marking committed up to index 875
1029 08:52:29.148514 (+ 3856us) raft_consensus.cc:1539] Triggering prepare for 281 ops
1029 08:52:29.148886 (+ 372us) write_op.cc:270] Start()
1029 08:52:29.148938 (+ 52us) write_op.cc:276] Timestamp: P: 1761727936485839 usec, L: 0
1029 08:52:29.168545 (+ 19607us) write_op.cc:270] Start()
1029 08:52:29.168606 (+ 61us) write_op.cc:276] Timestamp: P: 1761727936487054 usec, L: 0
1029 08:52:29.179364 (+ 10758us) write_op.cc:270] Start()
1029 08:52:29.179407 (+ 43us) write_op.cc:276] Timestamp: P: 1761727936489161 usec, L: 0
1029 08:52:29.184948 (+ 5541us) write_op.cc:270] Start()
1029 08:52:29.184983 (+ 35us) write_op.cc:276] Timestamp: P: 1761727936490008 usec, L: 0
1029 08:52:29.185899 (+ 916us) write_op.cc:270] Start()
1029 08:52:29.185931 (+ 32us) write_op.cc:276] Timestamp: P: 1761727936525708 usec, L: 0
1029 08:52:29.186697 (+ 766us) write_op.cc:270] Start()
1029 08:52:29.186719 (+ 22us) write_op.cc:276] Timestamp: P: 1761727936526853 usec, L: 0
1029 08:52:29.187474 (+ 755us) write_op.cc:270] Start()
1029 08:52:29.187515 (+ 41us) write_op.cc:276] Timestamp: P: 1761727936527825 usec, L: 0
1029 08:52:29.188338 (+ 823us) write_op.cc:270] Start()
1029 08:52:29.188362 (+ 24us) write_op.cc:276] Timestamp: P: 1761727936619792 usec, L: 0
1029 08:52:29.189102 (+ 740us) write_op.cc:270] Start()
1029 08:52:29.189125 (+ 23us) write_op.cc:276] Timestamp: P: 1761727936637396 usec, L: 0
1029 08:52:29.189942 (+ 817us) write_op.cc:270] Start()
1029 08:52:29.189977 (+ 35us) write_op.cc:276] Timestamp: P: 1761727936664872 usec, L: 0
1029 08:52:29.190898 (+ 921us) write_op.cc:270] Start()
1029 08:52:29.190922 (+ 24us) write_op.cc:276] Timestamp: P: 1761727936666505 usec, L: 0
1029 08:52:29.191536 (+ 614us) write_op.cc:270] Start()
1029 08:52:29.191558 (+ 22us) write_op.cc:276] Timestamp: P: 1761727936667386 usec, L: 0
1029 08:52:29.279043 (+ 87485us) write_op.cc:270] Start()
1029 08:52:29.279083 (+ 40us) write_op.cc:276] Timestamp: P: 1761727936668517 usec, L: 0
1029 08:52:29.279979 (+ 896us) write_op.cc:270] Start()
1029 08:52:29.280047 (+ 68us) write_op.cc:276] Timestamp: P: 1761727936669428 usec, L: 0
1029 08:52:29.280784 (+ 737us) write_op.cc:270] Start()
1029 08:52:29.280817 (+ 33us) write_op.cc:276] Timestamp: P: 1761727936671619 usec, L: 0
1029 08:52:29.289804 (+ 8987us) write_op.cc:270] Start()
1029 08:52:29.289837 (+ 33us) write_op.cc:276] Timestamp: P: 1761727936688275 usec, L: 0
1029 08:52:29.290402 (+ 565us) write_op.cc:270] Start()
1029 08:52:29.290437 (+ 35us) write_op.cc:276] Timestamp: P: 1761727936689429 usec, L: 0
1029 08:52:29.290951 (+ 514us) write_op.cc:270] Start()
1029 08:52:29.290972 (+ 21us) write_op.cc:276] Timestamp: P: 1761727936700371 usec, L: 0
1029 08:52:29.291417 (+ 445us) write_op.cc:270] Start()
1029 08:52:29.291438 (+ 21us) write_op.cc:276] Timestamp: P: 1761727936765111 usec, L: 0
1029 08:52:29.291932 (+ 494us) write_op.cc:270] Start()
1029 08:52:29.291955 (+ 23us) write_op.cc:276] Timestamp: P: 1761727936798104 usec, L: 0
1029 08:52:29.294944 (+ 2989us) write_op.cc:270] Start()
1029 08:52:29.294970 (+ 26us) write_op.cc:276] Timestamp: P: 1761727936801084 usec, L: 0
1029 08:52:29.295883 (+ 913us) write_op.cc:270] Start()
1029 08:52:29.295939 (+ 56us) write_op.cc:276] Timestamp: P: 1761727936801969 usec, L: 0
1029 08:52:29.298009 (+ 2070us) write_op.cc:270] Start()
1029 08:52:29.298035 (+ 26us) write_op.cc:276] Timestamp: P: 1761727936802704 usec, L: 0
1029 08:52:29.304804 (+ 6769us) write_op.cc:270] Start()
1029 08:52:29.304833 (+ 29us) write_op.cc:276] Timestamp: P: 1761727936835890 usec, L: 0
1029 08:52:29.305356 (+ 523us) write_op.cc:270] Start()
1029 08:52:29.305385 (+ 29us) write_op.cc:276] Timestamp: P: 1761727936837677 usec, L: 0
1029 08:52:29.306103 (+ 718us) write_op.cc:270] Start()
1029 08:52:29.306129 (+ 26us) write_op.cc:276] Timestamp: P: 1761727936842080 usec, L: 0
1029 08:52:29.312782 (+ 6653us) write_op.cc:270] Start()
1029 08:52:29.312909 (+ 127us) write_op.cc:276] Timestamp: P: 1761727936860852 usec, L: 0
1029 08:52:29.313725 (+ 816us) write_op.cc:270] Start()
1029 08:52:29.313765 (+ 40us) write_op.cc:276] Timestamp: P: 1761727936874760 usec, L: 0
1029 08:52:29.314268 (+ 503us) write_op.cc:270] Start()
1029 08:52:29.314292 (+ 24us) write_op.cc:276] Timestamp: P: 1761727936921474 usec, L: 0
1029 08:52:29.315589 (+ 1297us) write_op.cc:270] Start()
1029 08:52:29.315612 (+ 23us) write_op.cc:276] Timestamp: P: 1761727936949428 usec, L: 0
1029 08:52:29.316122 (+ 510us) write_op.cc:270] Start()
1029 08:52:29.316144 (+ 22us) write_op.cc:276] Timestamp: P: 1761727936956509 usec, L: 0
1029 08:52:29.318093 (+ 1949us) write_op.cc:270] Start()
1029 08:52:29.318116 (+ 23us) write_op.cc:276] Timestamp: P: 1761727936980296 usec, L: 0
1029 08:52:29.320504 (+ 2388us) write_op.cc:270] Start()
1029 08:52:29.320528 (+ 24us) write_op.cc:276] Timestamp: P: 1761727936982796 usec, L: 0
1029 08:52:29.321319 (+ 791us) write_op.cc:270] Start()
1029 08:52:29.321344 (+ 25us) write_op.cc:276] Timestamp: P: 1761727936992750 usec, L: 0
1029 08:52:29.322128 (+ 784us) write_op.cc:270] Start()
1029 08:52:29.322152 (+ 24us) write_op.cc:276] Timestamp: P: 1761727936995655 usec, L: 0
1029 08:52:29.322294 (+ 142us) write_op.cc:270] Start()
1029 08:52:29.322314 (+ 20us) write_op.cc:276] Timestamp: P: 1761727937064500 usec, L: 0
1029 08:52:29.322435 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.322471 (+ 36us) write_op.cc:276] Timestamp: P: 1761727937065651 usec, L: 0
1029 08:52:29.322590 (+ 119us) write_op.cc:270] Start()
1029 08:52:29.322607 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937066589 usec, L: 0
1029 08:52:29.322749 (+ 142us) write_op.cc:270] Start()
1029 08:52:29.322767 (+ 18us) write_op.cc:276] Timestamp: P: 1761727937073414 usec, L: 0
1029 08:52:29.322904 (+ 137us) write_op.cc:270] Start()
1029 08:52:29.322923 (+ 19us) write_op.cc:276] Timestamp: P: 1761727937074066 usec, L: 0
1029 08:52:29.323034 (+ 111us) write_op.cc:270] Start()
1029 08:52:29.323051 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937074550 usec, L: 0
1029 08:52:29.323186 (+ 135us) write_op.cc:270] Start()
1029 08:52:29.323205 (+ 19us) write_op.cc:276] Timestamp: P: 1761727937075054 usec, L: 0
1029 08:52:29.330578 (+ 7373us) write_op.cc:270] Start()
1029 08:52:29.330613 (+ 35us) write_op.cc:276] Timestamp: P: 1761727937094510 usec, L: 0
1029 08:52:29.330847 (+ 234us) write_op.cc:270] Start()
1029 08:52:29.330879 (+ 32us) write_op.cc:276] Timestamp: P: 1761727937125192 usec, L: 0
1029 08:52:29.331046 (+ 167us) write_op.cc:270] Start()
1029 08:52:29.331063 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937205604 usec, L: 0
1029 08:52:29.331175 (+ 112us) write_op.cc:270] Start()
1029 08:52:29.331191 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937232314 usec, L: 0
1029 08:52:29.331303 (+ 112us) write_op.cc:270] Start()
1029 08:52:29.331319 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937232967 usec, L: 0
1029 08:52:29.331452 (+ 133us) write_op.cc:270] Start()
1029 08:52:29.331469 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937233473 usec, L: 0
1029 08:52:29.331585 (+ 116us) write_op.cc:270] Start()
1029 08:52:29.331601 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937233979 usec, L: 0
1029 08:52:29.331824 (+ 223us) write_op.cc:270] Start()
1029 08:52:29.331844 (+ 20us) write_op.cc:276] Timestamp: P: 1761727937234500 usec, L: 0
1029 08:52:29.331957 (+ 113us) write_op.cc:270] Start()
1029 08:52:29.331974 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937235008 usec, L: 0
1029 08:52:29.332142 (+ 168us) write_op.cc:270] Start()
1029 08:52:29.332169 (+ 27us) write_op.cc:276] Timestamp: P: 1761727937235473 usec, L: 0
1029 08:52:29.332335 (+ 166us) write_op.cc:270] Start()
1029 08:52:29.332358 (+ 23us) write_op.cc:276] Timestamp: P: 1761727937235959 usec, L: 0
1029 08:52:29.332503 (+ 145us) write_op.cc:270] Start()
1029 08:52:29.332528 (+ 25us) write_op.cc:276] Timestamp: P: 1761727937389962 usec, L: 0
1029 08:52:29.332711 (+ 183us) write_op.cc:270] Start()
1029 08:52:29.332734 (+ 23us) write_op.cc:276] Timestamp: P: 1761727937392396 usec, L: 0
1029 08:52:29.332889 (+ 155us) write_op.cc:270] Start()
1029 08:52:29.332912 (+ 23us) write_op.cc:276] Timestamp: P: 1761727937465586 usec, L: 0
1029 08:52:29.333057 (+ 145us) write_op.cc:270] Start()
1029 08:52:29.333080 (+ 23us) write_op.cc:276] Timestamp: P: 1761727937529686 usec, L: 0
1029 08:52:29.333197 (+ 117us) write_op.cc:270] Start()
1029 08:52:29.333214 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937532284 usec, L: 0
1029 08:52:29.347217 (+ 14003us) write_op.cc:270] Start()
1029 08:52:29.347272 (+ 55us) write_op.cc:276] Timestamp: P: 1761727937580221 usec, L: 0
1029 08:52:29.347526 (+ 254us) write_op.cc:270] Start()
1029 08:52:29.347579 (+ 53us) write_op.cc:276] Timestamp: P: 1761727937583417 usec, L: 0
1029 08:52:29.347709 (+ 130us) write_op.cc:270] Start()
1029 08:52:29.347725 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937664975 usec, L: 0
1029 08:52:29.347828 (+ 103us) write_op.cc:270] Start()
1029 08:52:29.347843 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937700715 usec, L: 0
1029 08:52:29.347950 (+ 107us) write_op.cc:270] Start()
1029 08:52:29.347965 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937703391 usec, L: 0
1029 08:52:29.348096 (+ 131us) write_op.cc:270] Start()
1029 08:52:29.348112 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937731513 usec, L: 0
1029 08:52:29.348269 (+ 157us) write_op.cc:270] Start()
1029 08:52:29.348286 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937808925 usec, L: 0
1029 08:52:29.348412 (+ 126us) write_op.cc:270] Start()
1029 08:52:29.348427 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937809696 usec, L: 0
1029 08:52:29.348523 (+ 96us) write_op.cc:270] Start()
1029 08:52:29.348538 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937810207 usec, L: 0
1029 08:52:29.348635 (+ 97us) write_op.cc:270] Start()
1029 08:52:29.348649 (+ 14us) write_op.cc:276] Timestamp: P: 1761727937816709 usec, L: 0
1029 08:52:29.348766 (+ 117us) write_op.cc:270] Start()
1029 08:52:29.348782 (+ 16us) write_op.cc:276] Timestamp: P: 1761727937822591 usec, L: 0
1029 08:52:29.348882 (+ 100us) write_op.cc:270] Start()
1029 08:52:29.348896 (+ 14us) write_op.cc:276] Timestamp: P: 1761727937823190 usec, L: 0
1029 08:52:29.349007 (+ 111us) write_op.cc:270] Start()
1029 08:52:29.349022 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937823675 usec, L: 0
1029 08:52:29.349136 (+ 114us) write_op.cc:270] Start()
1029 08:52:29.349150 (+ 14us) write_op.cc:276] Timestamp: P: 1761727937824273 usec, L: 0
1029 08:52:29.349252 (+ 102us) write_op.cc:270] Start()
1029 08:52:29.349267 (+ 15us) write_op.cc:276] Timestamp: P: 1761727937824756 usec, L: 0
1029 08:52:29.349476 (+ 209us) write_op.cc:270] Start()
1029 08:52:29.349503 (+ 27us) write_op.cc:276] Timestamp: P: 1761727937852076 usec, L: 0
1029 08:52:29.349654 (+ 151us) write_op.cc:270] Start()
1029 08:52:29.349686 (+ 32us) write_op.cc:276] Timestamp: P: 1761727937877743 usec, L: 0
1029 08:52:29.349825 (+ 139us) write_op.cc:270] Start()
1029 08:52:29.349844 (+ 19us) write_op.cc:276] Timestamp: P: 1761727937883599 usec, L: 0
1029 08:52:29.349960 (+ 116us) write_op.cc:270] Start()
1029 08:52:29.349974 (+ 14us) write_op.cc:276] Timestamp: P: 1761727937886808 usec, L: 0
1029 08:52:29.350093 (+ 119us) write_op.cc:270] Start()
1029 08:52:29.350110 (+ 17us) write_op.cc:276] Timestamp: P: 1761727937887413 usec, L: 0
1029 08:52:29.350212 (+ 102us) write_op.cc:270] Start()
1029 08:52:29.350245 (+ 33us) write_op.cc:276] Timestamp: P: 1761727937888022 usec, L: 0
1029 08:52:29.350373 (+ 128us) write_op.cc:270] Start()
1029 08:52:29.350386 (+ 13us) write_op.cc:276] Timestamp: P: 1761727938038619 usec, L: 0
1029 08:52:29.350478 (+ 92us) write_op.cc:270] Start()
1029 08:52:29.350494 (+ 16us) write_op.cc:276] Timestamp: P: 1761727938040612 usec, L: 0
1029 08:52:29.350607 (+ 113us) write_op.cc:270] Start()
1029 08:52:29.350623 (+ 16us) write_op.cc:276] Timestamp: P: 1761727938059837 usec, L: 0
1029 08:52:29.350749 (+ 126us) write_op.cc:270] Start()
1029 08:52:29.350778 (+ 29us) write_op.cc:276] Timestamp: P: 1761727938060538 usec, L: 0
1029 08:52:29.350888 (+ 110us) write_op.cc:270] Start()
1029 08:52:29.350902 (+ 14us) write_op.cc:276] Timestamp: P: 1761727938061090 usec, L: 0
1029 08:52:29.367634 (+ 16732us) write_op.cc:270] Start()
1029 08:52:29.367687 (+ 53us) write_op.cc:276] Timestamp: P: 1761727938091603 usec, L: 0
1029 08:52:29.367912 (+ 225us) write_op.cc:270] Start()
1029 08:52:29.367935 (+ 23us) write_op.cc:276] Timestamp: P: 1761727938092289 usec, L: 0
1029 08:52:29.368092 (+ 157us) write_op.cc:270] Start()
1029 08:52:29.368113 (+ 21us) write_op.cc:276] Timestamp: P: 1761727938092827 usec, L: 0
1029 08:52:29.368239 (+ 126us) write_op.cc:270] Start()
1029 08:52:29.368305 (+ 66us) write_op.cc:276] Timestamp: P: 1761727938101196 usec, L: 0
1029 08:52:29.368440 (+ 135us) write_op.cc:270] Start()
1029 08:52:29.368460 (+ 20us) write_op.cc:276] Timestamp: P: 1761727938121112 usec, L: 0
1029 08:52:29.368622 (+ 162us) write_op.cc:270] Start()
1029 08:52:29.368644 (+ 22us) write_op.cc:276] Timestamp: P: 1761727938156786 usec, L: 0
1029 08:52:29.368823 (+ 179us) write_op.cc:270] Start()
1029 08:52:29.368846 (+ 23us) write_op.cc:276] Timestamp: P: 1761727938157451 usec, L: 0
1029 08:52:29.368997 (+ 151us) write_op.cc:270] Start()
1029 08:52:29.369021 (+ 24us) write_op.cc:276] Timestamp: P: 1761727938184896 usec, L: 0
1029 08:52:29.369190 (+ 169us) write_op.cc:270] Start()
1029 08:52:29.369218 (+ 28us) write_op.cc:276] Timestamp: P: 1761727938186014 usec, L: 0
1029 08:52:29.369384 (+ 166us) write_op.cc:270] Start()
1029 08:52:29.369408 (+ 24us) write_op.cc:276] Timestamp: P: 1761727938238405 usec, L: 0
1029 08:52:29.369746 (+ 338us) write_op.cc:270] Start()
1029 08:52:29.369771 (+ 25us) write_op.cc:276] Timestamp: P: 1761727938288724 usec, L: 0
1029 08:52:29.370289 (+ 518us) write_op.cc:270] Start()
1029 08:52:29.370318 (+ 29us) write_op.cc:276] Timestamp: P: 1761727938387253 usec, L: 0
1029 08:52:29.370484 (+ 166us) write_op.cc:270] Start()
1029 08:52:29.370509 (+ 25us) write_op.cc:276] Timestamp: P: 1761727938401318 usec, L: 0
1029 08:52:29.370670 (+ 161us) write_op.cc:270] Start()
1029 08:52:29.370704 (+ 34us) write_op.cc:276] Timestamp: P: 1761727938405398 usec, L: 0
1029 08:52:29.370878 (+ 174us) write_op.cc:270] Start()
1029 08:52:29.370894 (+ 16us) write_op.cc:276] Timestamp: P: 1761727938409390 usec, L: 0
1029 08:52:29.371020 (+ 126us) write_op.cc:270] Start()
1029 08:52:29.371041 (+ 21us) write_op.cc:276] Timestamp: P: 1761727938414923 usec, L: 0
1029 08:52:29.371155 (+ 114us) write_op.cc:270] Start()
1029 08:52:29.371182 (+ 27us) write_op.cc:276] Timestamp: P: 1761727938415593 usec, L: 0
1029 08:52:29.371311 (+ 129us) write_op.cc:270] Start()
1029 08:52:29.371330 (+ 19us) write_op.cc:276] Timestamp: P: 1761727938416273 usec, L: 0
1029 08:52:29.371451 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.371470 (+ 19us) write_op.cc:276] Timestamp: P: 1761727938417542 usec, L: 0
1029 08:52:29.371601 (+ 131us) write_op.cc:270] Start()
1029 08:52:29.371619 (+ 18us) write_op.cc:276] Timestamp: P: 1761727938418471 usec, L: 0
1029 08:52:29.371753 (+ 134us) write_op.cc:270] Start()
1029 08:52:29.371772 (+ 19us) write_op.cc:276] Timestamp: P: 1761727938433364 usec, L: 0
1029 08:52:29.371897 (+ 125us) write_op.cc:270] Start()
1029 08:52:29.371915 (+ 18us) write_op.cc:276] Timestamp: P: 1761727938589181 usec, L: 0
1029 08:52:29.377127 (+ 5212us) write_op.cc:270] Start()
1029 08:52:29.377169 (+ 42us) write_op.cc:276] Timestamp: P: 1761727938696800 usec, L: 0
1029 08:52:29.377330 (+ 161us) write_op.cc:270] Start()
1029 08:52:29.377347 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938699538 usec, L: 0
1029 08:52:29.377468 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.377487 (+ 19us) write_op.cc:276] Timestamp: P: 1761727938700491 usec, L: 0
1029 08:52:29.377614 (+ 127us) write_op.cc:270] Start()
1029 08:52:29.377631 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938701072 usec, L: 0
1029 08:52:29.377749 (+ 118us) write_op.cc:270] Start()
1029 08:52:29.377766 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938707673 usec, L: 0
1029 08:52:29.377877 (+ 111us) write_op.cc:270] Start()
1029 08:52:29.377894 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938711796 usec, L: 0
1029 08:52:29.377996 (+ 102us) write_op.cc:270] Start()
1029 08:52:29.378011 (+ 15us) write_op.cc:276] Timestamp: P: 1761727938712765 usec, L: 0
1029 08:52:29.378118 (+ 107us) write_op.cc:270] Start()
1029 08:52:29.378134 (+ 16us) write_op.cc:276] Timestamp: P: 1761727938756701 usec, L: 0
1029 08:52:29.378249 (+ 115us) write_op.cc:270] Start()
1029 08:52:29.378265 (+ 16us) write_op.cc:276] Timestamp: P: 1761727938765580 usec, L: 0
1029 08:52:29.378363 (+ 98us) write_op.cc:270] Start()
1029 08:52:29.378378 (+ 15us) write_op.cc:276] Timestamp: P: 1761727938839163 usec, L: 0
1029 08:52:29.378486 (+ 108us) write_op.cc:270] Start()
1029 08:52:29.378503 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938866723 usec, L: 0
1029 08:52:29.378620 (+ 117us) write_op.cc:270] Start()
1029 08:52:29.378637 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938928243 usec, L: 0
1029 08:52:29.378758 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.378775 (+ 17us) write_op.cc:276] Timestamp: P: 1761727938949780 usec, L: 0
1029 08:52:29.378926 (+ 151us) write_op.cc:270] Start()
1029 08:52:29.378944 (+ 18us) write_op.cc:276] Timestamp: P: 1761727938991737 usec, L: 0
1029 08:52:29.379047 (+ 103us) write_op.cc:270] Start()
1029 08:52:29.379062 (+ 15us) write_op.cc:276] Timestamp: P: 1761727938992838 usec, L: 0
1029 08:52:29.379209 (+ 147us) write_op.cc:270] Start()
1029 08:52:29.379227 (+ 18us) write_op.cc:276] Timestamp: P: 1761727938993741 usec, L: 0
1029 08:52:29.379364 (+ 137us) write_op.cc:270] Start()
1029 08:52:29.379382 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939117927 usec, L: 0
1029 08:52:29.379496 (+ 114us) write_op.cc:270] Start()
1029 08:52:29.379514 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939118998 usec, L: 0
1029 08:52:29.379653 (+ 139us) write_op.cc:270] Start()
1029 08:52:29.379669 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939120851 usec, L: 0
1029 08:52:29.379787 (+ 118us) write_op.cc:270] Start()
1029 08:52:29.379803 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939121830 usec, L: 0
1029 08:52:29.379942 (+ 139us) write_op.cc:270] Start()
1029 08:52:29.379957 (+ 15us) write_op.cc:276] Timestamp: P: 1761727939122784 usec, L: 0
1029 08:52:29.380186 (+ 229us) write_op.cc:270] Start()
1029 08:52:29.380207 (+ 21us) write_op.cc:276] Timestamp: P: 1761727939243421 usec, L: 0
1029 08:52:29.380972 (+ 765us) write_op.cc:270] Start()
1029 08:52:29.380999 (+ 27us) write_op.cc:276] Timestamp: P: 1761727939244943 usec, L: 0
1029 08:52:29.381150 (+ 151us) write_op.cc:270] Start()
1029 08:52:29.381168 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939245882 usec, L: 0
1029 08:52:29.381278 (+ 110us) write_op.cc:270] Start()
1029 08:52:29.381313 (+ 35us) write_op.cc:276] Timestamp: P: 1761727939246750 usec, L: 0
1029 08:52:29.381423 (+ 110us) write_op.cc:270] Start()
1029 08:52:29.381440 (+ 17us) write_op.cc:276] Timestamp: P: 1761727939333090 usec, L: 0
1029 08:52:29.381544 (+ 104us) write_op.cc:270] Start()
1029 08:52:29.381560 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939335895 usec, L: 0
1029 08:52:29.381685 (+ 125us) write_op.cc:270] Start()
1029 08:52:29.381702 (+ 17us) write_op.cc:276] Timestamp: P: 1761727939336957 usec, L: 0
1029 08:52:29.381804 (+ 102us) write_op.cc:270] Start()
1029 08:52:29.381820 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939337946 usec, L: 0
1029 08:52:29.381926 (+ 106us) write_op.cc:270] Start()
1029 08:52:29.381942 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939444981 usec, L: 0
1029 08:52:29.382042 (+ 100us) write_op.cc:270] Start()
1029 08:52:29.382057 (+ 15us) write_op.cc:276] Timestamp: P: 1761727939445985 usec, L: 0
1029 08:52:29.382162 (+ 105us) write_op.cc:270] Start()
1029 08:52:29.382178 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939446837 usec, L: 0
1029 08:52:29.382291 (+ 113us) write_op.cc:270] Start()
1029 08:52:29.382307 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939448443 usec, L: 0
1029 08:52:29.382432 (+ 125us) write_op.cc:270] Start()
1029 08:52:29.382450 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939449418 usec, L: 0
1029 08:52:29.382574 (+ 124us) write_op.cc:270] Start()
1029 08:52:29.382590 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939566578 usec, L: 0
1029 08:52:29.382711 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.382727 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939567640 usec, L: 0
1029 08:52:29.382827 (+ 100us) write_op.cc:270] Start()
1029 08:52:29.382843 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939568608 usec, L: 0
1029 08:52:29.382954 (+ 111us) write_op.cc:270] Start()
1029 08:52:29.382969 (+ 15us) write_op.cc:276] Timestamp: P: 1761727939569435 usec, L: 0
1029 08:52:29.383062 (+ 93us) write_op.cc:270] Start()
1029 08:52:29.383078 (+ 16us) write_op.cc:276] Timestamp: P: 1761727939570231 usec, L: 0
1029 08:52:29.383229 (+ 151us) write_op.cc:270] Start()
1029 08:52:29.383253 (+ 24us) write_op.cc:276] Timestamp: P: 1761727939746625 usec, L: 0
1029 08:52:29.383394 (+ 141us) write_op.cc:270] Start()
1029 08:52:29.383414 (+ 20us) write_op.cc:276] Timestamp: P: 1761727939747927 usec, L: 0
1029 08:52:29.383567 (+ 153us) write_op.cc:270] Start()
1029 08:52:29.383592 (+ 25us) write_op.cc:276] Timestamp: P: 1761727939892432 usec, L: 0
1029 08:52:29.383749 (+ 157us) write_op.cc:270] Start()
1029 08:52:29.383770 (+ 21us) write_op.cc:276] Timestamp: P: 1761727939893978 usec, L: 0
1029 08:52:29.383944 (+ 174us) write_op.cc:270] Start()
1029 08:52:29.383965 (+ 21us) write_op.cc:276] Timestamp: P: 1761727939895300 usec, L: 0
1029 08:52:29.401575 (+ 17610us) write_op.cc:270] Start()
1029 08:52:29.401627 (+ 52us) write_op.cc:276] Timestamp: P: 1761727939896710 usec, L: 0
1029 08:52:29.401856 (+ 229us) write_op.cc:270] Start()
1029 08:52:29.401879 (+ 23us) write_op.cc:276] Timestamp: P: 1761727939897834 usec, L: 0
1029 08:52:29.402007 (+ 128us) write_op.cc:270] Start()
1029 08:52:29.402025 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939943480 usec, L: 0
1029 08:52:29.402154 (+ 129us) write_op.cc:270] Start()
1029 08:52:29.402173 (+ 19us) write_op.cc:276] Timestamp: P: 1761727939965219 usec, L: 0
1029 08:52:29.402316 (+ 143us) write_op.cc:270] Start()
1029 08:52:29.402343 (+ 27us) write_op.cc:276] Timestamp: P: 1761727939972072 usec, L: 0
1029 08:52:29.408633 (+ 6290us) write_op.cc:270] Start()
1029 08:52:29.408665 (+ 32us) write_op.cc:276] Timestamp: P: 1761727939974008 usec, L: 0
1029 08:52:29.408862 (+ 197us) write_op.cc:270] Start()
1029 08:52:29.408885 (+ 23us) write_op.cc:276] Timestamp: P: 1761727939979059 usec, L: 0
1029 08:52:29.409012 (+ 127us) write_op.cc:270] Start()
1029 08:52:29.409031 (+ 19us) write_op.cc:276] Timestamp: P: 1761727939980811 usec, L: 0
1029 08:52:29.409145 (+ 114us) write_op.cc:270] Start()
1029 08:52:29.409163 (+ 18us) write_op.cc:276] Timestamp: P: 1761727939990225 usec, L: 0
1029 08:52:29.409305 (+ 142us) write_op.cc:270] Start()
1029 08:52:29.409323 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940006199 usec, L: 0
1029 08:52:29.409449 (+ 126us) write_op.cc:270] Start()
1029 08:52:29.409469 (+ 20us) write_op.cc:276] Timestamp: P: 1761727940035322 usec, L: 0
1029 08:52:29.409599 (+ 130us) write_op.cc:270] Start()
1029 08:52:29.409617 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940038304 usec, L: 0
1029 08:52:29.409749 (+ 132us) write_op.cc:270] Start()
1029 08:52:29.409770 (+ 21us) write_op.cc:276] Timestamp: P: 1761727940086471 usec, L: 0
1029 08:52:29.409891 (+ 121us) write_op.cc:270] Start()
1029 08:52:29.409909 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940112882 usec, L: 0
1029 08:52:29.410028 (+ 119us) write_op.cc:270] Start()
1029 08:52:29.410046 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940120090 usec, L: 0
1029 08:52:29.410179 (+ 133us) write_op.cc:270] Start()
1029 08:52:29.410200 (+ 21us) write_op.cc:276] Timestamp: P: 1761727940120934 usec, L: 0
1029 08:52:29.410353 (+ 153us) write_op.cc:270] Start()
1029 08:52:29.410441 (+ 88us) write_op.cc:276] Timestamp: P: 1761727940128329 usec, L: 0
1029 08:52:29.410619 (+ 178us) write_op.cc:270] Start()
1029 08:52:29.410638 (+ 19us) write_op.cc:276] Timestamp: P: 1761727940129249 usec, L: 0
1029 08:52:29.410774 (+ 136us) write_op.cc:270] Start()
1029 08:52:29.410791 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940185352 usec, L: 0
1029 08:52:29.410948 (+ 157us) write_op.cc:270] Start()
1029 08:52:29.410976 (+ 28us) write_op.cc:276] Timestamp: P: 1761727940204868 usec, L: 0
1029 08:52:29.411146 (+ 170us) write_op.cc:270] Start()
1029 08:52:29.411170 (+ 24us) write_op.cc:276] Timestamp: P: 1761727940209756 usec, L: 0
1029 08:52:29.411288 (+ 118us) write_op.cc:270] Start()
1029 08:52:29.411305 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940236927 usec, L: 0
1029 08:52:29.411441 (+ 136us) write_op.cc:270] Start()
1029 08:52:29.411459 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940254521 usec, L: 0
1029 08:52:29.411602 (+ 143us) write_op.cc:270] Start()
1029 08:52:29.411620 (+ 18us) write_op.cc:276] Timestamp: P: 1761727940256772 usec, L: 0
1029 08:52:29.411747 (+ 127us) write_op.cc:270] Start()
1029 08:52:29.411764 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940265081 usec, L: 0
1029 08:52:29.415568 (+ 3804us) write_op.cc:270] Start()
1029 08:52:29.415593 (+ 25us) write_op.cc:276] Timestamp: P: 1761727940268795 usec, L: 0
1029 08:52:29.415729 (+ 136us) write_op.cc:270] Start()
1029 08:52:29.415746 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940457571 usec, L: 0
1029 08:52:29.415855 (+ 109us) write_op.cc:270] Start()
1029 08:52:29.415871 (+ 16us) write_op.cc:276] Timestamp: P: 1761727940463058 usec, L: 0
1029 08:52:29.415985 (+ 114us) write_op.cc:270] Start()
1029 08:52:29.416000 (+ 15us) write_op.cc:276] Timestamp: P: 1761727940463809 usec, L: 0
1029 08:52:29.417168 (+ 1168us) write_op.cc:270] Start()
1029 08:52:29.417457 (+ 289us) write_op.cc:276] Timestamp: P: 1761727940464417 usec, L: 0
1029 08:52:29.417620 (+ 163us) write_op.cc:270] Start()
1029 08:52:29.417640 (+ 20us) write_op.cc:276] Timestamp: P: 1761727940464994 usec, L: 0
1029 08:52:29.418074 (+ 434us) write_op.cc:270] Start()
1029 08:52:29.418096 (+ 22us) write_op.cc:276] Timestamp: P: 1761727940518963 usec, L: 0
1029 08:52:29.418564 (+ 468us) write_op.cc:270] Start()
1029 08:52:29.418588 (+ 24us) write_op.cc:276] Timestamp: P: 1761727940615873 usec, L: 0
1029 08:52:29.419558 (+ 970us) write_op.cc:270] Start()
1029 08:52:29.419585 (+ 27us) write_op.cc:276] Timestamp: P: 1761727940626821 usec, L: 0
1029 08:52:29.423329 (+ 3744us) write_op.cc:270] Start()
1029 08:52:29.423365 (+ 36us) write_op.cc:276] Timestamp: P: 1761727940633256 usec, L: 0
1029 08:52:29.423554 (+ 189us) write_op.cc:270] Start()
1029 08:52:29.423571 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940659710 usec, L: 0
1029 08:52:29.423707 (+ 136us) write_op.cc:270] Start()
1029 08:52:29.423724 (+ 17us) write_op.cc:276] Timestamp: P: 1761727940662126 usec, L: 0
1029 08:52:29.423830 (+ 106us) write_op.cc:270] Start()
1029 08:52:29.423845 (+ 15us) write_op.cc:276] Timestamp: P: 1761727940682185 usec, L: 0
1029 08:52:29.423935 (+ 90us) write_op.cc:270] Start()
1029 08:52:29.423949 (+ 14us) write_op.cc:276] Timestamp: P: 1761727940696402 usec, L: 0
1029 08:52:29.424074 (+ 125us) write_op.cc:270] Start()
1029 08:52:29.424090 (+ 16us) write_op.cc:276] Timestamp: P: 1761727940730208 usec, L: 0
1029 08:52:29.424194 (+ 104us) write_op.cc:270] Start()
1029 08:52:29.424208 (+ 14us) write_op.cc:276] Timestamp: P: 1761727940761832 usec, L: 0
1029 08:52:29.450047 (+ 25839us) write_op.cc:270] Start()
1029 08:52:29.450107 (+ 60us) write_op.cc:276] Timestamp: P: 1761727940816223 usec, L: 0
1029 08:52:29.450342 (+ 235us) write_op.cc:270] Start()
1029 08:52:29.450371 (+ 29us) write_op.cc:276] Timestamp: P: 1761727940824520 usec, L: 0
1029 08:52:29.450518 (+ 147us) write_op.cc:270] Start()
1029 08:52:29.450540 (+ 22us) write_op.cc:276] Timestamp: P: 1761727940830811 usec, L: 0
1029 08:52:29.450673 (+ 133us) write_op.cc:270] Start()
1029 08:52:29.451435 (+ 762us) write_op.cc:276] Timestamp: P: 1761727940845529 usec, L: 0
1029 08:52:29.451624 (+ 189us) write_op.cc:270] Start()
1029 08:52:29.451646 (+ 22us) write_op.cc:276] Timestamp: P: 1761727940893811 usec, L: 0
1029 08:52:29.451788 (+ 142us) write_op.cc:270] Start()
1029 08:52:29.451807 (+ 19us) write_op.cc:276] Timestamp: P: 17617
I20251029 08:52:30.147334 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1112) took 2486 ms. Trace:
I20251029 08:52:30.151566 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f718bd0f00004f20a89d028a2fa8e6b9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40435 } } peers { permanent_uuid: "85c8cdf28bda444d851fb50588400173" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38825 } } peers { permanent_uuid: "8395ff5ec92c4bbda9bc6641a9fbbb6b" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42135 } } }
I20251029 08:52:30.151760 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b [sys.catalog]: This master's current role is: LEADER
W20251029 08:52:30.152783 581 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f4cb38b4023242fa8e85356dd4cccd70 (table default.Table_4949f864d88f46d6b3e1ef34bbfc5418 [id=6cfbfe11dc1440cea62e314f0048743a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f4cb38b4023242fa8e85356dd4cccd70 consensus state has no leader
I20251029 08:52:30.152962 1176 rpcz_store.cc:276] 1029 08:52:27.660832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.660907 (+ 75us) service_pool.cc:225] Handling call
1029 08:52:30.076560 (+2415653us) ts_tablet_manager.cc:1151] Deleting tablet 913fb69271e14fe3a8d0b29d0db0dc8e
1029 08:52:30.076569 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.102480 (+ 25911us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.133005 (+ 30525us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.147315 (+ 14310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415476}
I20251029 08:52:30.153847 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 913fb69271e14fe3a8d0b29d0db0dc8e (table default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]) successfully deleted
I20251029 08:52:30.154287 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1109) took 2493 ms. Trace:
I20251029 08:52:30.155467 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a85fb70d0d2b44f6b4912645f18e8150" } modify_external_catalogs: true
I20251029 08:52:30.155607 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_a85fb70d0d2b44f6b4912645f18e8150" } modify_external_catalogs: true
I20251029 08:52:30.164007 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98e2b0275a064384b92201c42be8a086" } modify_external_catalogs: true
I20251029 08:52:30.164269 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_98e2b0275a064384b92201c42be8a086" } modify_external_catalogs: true
I20251029 08:52:30.144394 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cd2e9a95b6c40be9e17397a313d249b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.164741 1176 rpcz_store.cc:276] 1029 08:52:27.660587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.664076 (+ 3489us) service_pool.cc:225] Handling call
1029 08:52:30.154150 (+2490074us) ts_tablet_manager.cc:1151] Deleting tablet 21d409c2e24947beb99351da46a5aef9
1029 08:52:30.154161 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.154269 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2489911}
W20251029 08:52:30.165216 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 21d409c2e24947beb99351da46a5aef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d409c2e24947beb99351da46a5aef9 already in progress: creating tablet
I20251029 08:52:30.165269 1179 ts_tablet_manager.cc:1929] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.165408 1179 log.cc:1199] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:30.165659 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1115) took 2474 ms. Trace:
I20251029 08:52:30.165851 1179 ts_tablet_manager.cc:1950] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.165776 1176 rpcz_store.cc:276] 1029 08:52:27.691144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.691181 (+ 37us) service_pool.cc:225] Handling call
1029 08:52:30.165513 (+2474332us) ts_tablet_manager.cc:1151] Deleting tablet 1fb9c8c7286149128c2741635c242341
1029 08:52:30.165524 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.165637 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474195}
W20251029 08:52:30.166231 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 1fb9c8c7286149128c2741635c242341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb9c8c7286149128c2741635c242341 already in progress: creating tablet
I20251029 08:52:30.166788 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1083) took 2897 ms. Trace:
I20251029 08:52:30.166913 1179 rpcz_store.cc:276] 1029 08:52:27.269442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.270203 (+ 761us) service_pool.cc:225] Handling call
1029 08:52:30.141349 (+2871146us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:30.141359 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.153315 (+ 11956us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.165159 (+ 11844us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.166766 (+ 1607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871002}
I20251029 08:52:30.167357 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (table default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]) successfully deleted
I20251029 08:52:30.167683 1179 tablet_replica.cc:333] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.167941 1179 raft_consensus.cc:2243] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:30.168344 1179 raft_consensus.cc:2272] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.168426 1176 tablet_replica.cc:333] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.168601 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac09bb4502e440368633bbc90e9ad2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.168623 1176 raft_consensus.cc:2243] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.168794 1176 raft_consensus.cc:2272] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:30.169320 996 leader_election.cc:343] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.169427 996 leader_election.cc:304] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.169823 2707 raft_consensus.cc:2749] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.170333 1179 ts_tablet_manager.cc:1916] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.172242 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5525b69c8a4450b6d6571d39abb079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.172444 1183 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.173341 1176 ts_tablet_manager.cc:1916] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.174093 1183 tablet_bootstrap.cc:654] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.174981 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac8d52f6e954e96aa30f20c33d5d81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.179883 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213fc61402784f19a5287fdb57719616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.187165 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 3ee83f5526f74ba2a344edd97db94050
I20251029 08:52:30.188292 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ee83f5526f74ba2a344edd97db94050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:30.188576 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3990e2983ffe4f1ca26f68875416a72a has no consensus state
I20251029 08:52:30.189651 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e27701fbf5b44e90ba33bed99fdc4d9b
I20251029 08:52:30.190606 757 tablet_service.cc:1552] Processing DeleteTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:30.190697 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2bd3e1dcd8d04e13b990a8f064f69ce3 consensus state has no leader
I20251029 08:52:30.190871 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.191336 2786 raft_consensus.cc:493] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.191473 2786 raft_consensus.cc:515] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.191751 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9e44e56554574d5abb67d08d88218152
I20251029 08:52:30.192184 2786 leader_election.cc:290] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.192493 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e44e56554574d5abb67d08d88218152 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.192973 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb9ab76dcfe486e89bbe9391cab56f9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:30.193460 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e44e56554574d5abb67d08d88218152 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:30.193604 720 leader_election.cc:343] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.197118 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ee83f5526f74ba2a344edd97db94050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.198415 590 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_4498572fb2e34861a6de39a0b65d120f [id=8b155393505643e9b1b66a0a8ac5e4f2] alter complete (version 6)
I20251029 08:52:30.193174 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb9ab76dcfe486e89bbe9391cab56f9" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.200302 895 tablet_service.cc:1552] Processing DeleteTablet for tablet e27701fbf5b44e90ba33bed99fdc4d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.200377 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e44e56554574d5abb67d08d88218152 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
W20251029 08:52:30.200598 718 leader_election.cc:343] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.200713 718 leader_election.cc:304] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.201297 2786 raft_consensus.cc:2749] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.202311 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ee83f5526f74ba2a344edd97db94050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.206348 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f3d319733949430482ae8e14a2129b7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.207055 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3d319733949430482ae8e14a2129b7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.211617 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_43aa92e7fba74d91ad0b832e5f5d6592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.212157 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43aa92e7fba74d91ad0b832e5f5d6592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.218590 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_301169b3168b41b789340c0724a5afcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.219143 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_301169b3168b41b789340c0724a5afcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.219401 1177 tablet_bootstrap.cc:492] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:30.219542 1177 ts_tablet_manager.cc:1403] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.497s user 0.012s sys 0.035s
I20251029 08:52:30.220407 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f330ad2e9ac74e1c8ec0b43d0405a238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.220853 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f330ad2e9ac74e1c8ec0b43d0405a238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.221925 1179 ts_tablet_manager.cc:1929] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.221884 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_0c8626ad4f874c99842ee20fc62cf6c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.222028 1179 log.cc:1199] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/aa0ef904c2fb4dfeadbf56ae659c4e20
W20251029 08:52:30.222335 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c8626ad4f874c99842ee20fc62cf6c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.222364 1179 ts_tablet_manager.cc:1950] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.225945 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1084) took 2955 ms. Trace:
I20251029 08:52:30.226084 1179 rpcz_store.cc:276] 1029 08:52:27.270553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.270727 (+ 174us) service_pool.cc:225] Handling call
1029 08:52:30.167625 (+2896898us) ts_tablet_manager.cc:1151] Deleting tablet aa0ef904c2fb4dfeadbf56ae659c4e20
1029 08:52:30.167647 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.183779 (+ 16132us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.221806 (+ 38027us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.225918 (+ 4112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896763}
I20251029 08:52:30.228274 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (table default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]) successfully deleted
I20251029 08:52:30.228670 1179 tablet_replica.cc:333] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.228924 1179 raft_consensus.cc:2243] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.229166 1179 raft_consensus.cc:2272] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.229805 1179 ts_tablet_manager.cc:1916] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.233031 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cdaa068607404571bc2534b6ad0a3b6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.233621 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdaa068607404571bc2534b6ad0a3b6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.235852 1177 raft_consensus.cc:359] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.236060 1177 raft_consensus.cc:385] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.236114 1177 raft_consensus.cc:740] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.236296 1177 consensus_queue.cc:260] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.236690 1177 ts_tablet_manager.cc:1434] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20251029 08:52:30.236953 1177 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:30.238365 1177 tablet_bootstrap.cc:654] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.243254 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_25119faaae7141a8b57594132e159677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.243916 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25119faaae7141a8b57594132e159677 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.246610 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_702694262c424e65a9c1319f5c82eba3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.247087 1179 ts_tablet_manager.cc:1929] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.247202 1179 log.cc:1199] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4e213713757949caa3f504f70026b7b5
W20251029 08:52:30.247203 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_702694262c424e65a9c1319f5c82eba3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.247597 1179 ts_tablet_manager.cc:1950] T 4e213713757949caa3f504f70026b7b5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.248525 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1085) took 2977 ms. Trace:
I20251029 08:52:30.248808 1179 rpcz_store.cc:276] 1029 08:52:27.271042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.271215 (+ 173us) service_pool.cc:225] Handling call
1029 08:52:30.228597 (+2957382us) ts_tablet_manager.cc:1151] Deleting tablet 4e213713757949caa3f504f70026b7b5
1029 08:52:30.228609 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.236832 (+ 8223us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.246965 (+ 10133us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.248496 (+ 1531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2957243}
I20251029 08:52:30.249454 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4e213713757949caa3f504f70026b7b5 (table default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]) successfully deleted
I20251029 08:52:30.249979 1179 tablet_replica.cc:333] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.250211 1179 raft_consensus.cc:2243] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.250437 1179 raft_consensus.cc:2272] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.251094 1179 ts_tablet_manager.cc:1916] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.257294 1176 ts_tablet_manager.cc:1929] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:30.257476 1176 log.cc:1199] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/0cc95257c937454aa6cd55b5a9394a56
I20251029 08:52:30.257859 1176 ts_tablet_manager.cc:1950] T 0cc95257c937454aa6cd55b5a9394a56 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.258684 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1113) took 2567 ms. Trace:
I20251029 08:52:30.258791 1176 rpcz_store.cc:276] 1029 08:52:27.690888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.691210 (+ 322us) service_pool.cc:225] Handling call
1029 08:52:30.168373 (+2477163us) ts_tablet_manager.cc:1151] Deleting tablet 0cc95257c937454aa6cd55b5a9394a56
1029 08:52:30.168384 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.182814 (+ 14430us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.205127 (+ 22313us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.258655 (+ 53528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2476998}
I20251029 08:52:30.259331 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]) successfully deleted
I20251029 08:52:30.272490 1179 ts_tablet_manager.cc:1929] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.272745 1179 log.cc:1199] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/09968d93203140fa9201928c87702ee0
I20251029 08:52:30.273157 1179 ts_tablet_manager.cc:1950] T 09968d93203140fa9201928c87702ee0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.275151 1176 tablet_replica.cc:333] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.275363 1176 raft_consensus.cc:2243] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.275524 1176 raft_consensus.cc:2272] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.276074 1176 ts_tablet_manager.cc:1916] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.277077 595 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8832b446a68d4b7eae0c0974175a1ca4
I20251029 08:52:30.275166 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1086) took 3003 ms. Trace:
I20251029 08:52:30.277386 1179 rpcz_store.cc:276] 1029 08:52:27.271541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.271702 (+ 161us) service_pool.cc:225] Handling call
1029 08:52:30.249918 (+2978216us) ts_tablet_manager.cc:1151] Deleting tablet 09968d93203140fa9201928c87702ee0
1029 08:52:30.249930 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.258886 (+ 8956us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.272357 (+ 13471us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.275147 (+ 2790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978070}
I20251029 08:52:30.277520 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bce55107e4124f29a2e90ebc9781adae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.278084 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1090) took 2978 ms. Trace:
W20251029 08:52:30.278159 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bce55107e4124f29a2e90ebc9781adae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.278162 1179 rpcz_store.cc:276] 1029 08:52:27.299626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.299671 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:30.277955 (+2978284us) ts_tablet_manager.cc:1151] Deleting tablet f744556dccf348fd8b92071f224730aa
1029 08:52:30.277967 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.278066 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2974108}
W20251029 08:52:30.279407 861 consensus_peers.cc:597] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cc95257c937454aa6cd55b5a9394a56. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.279438 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0adba75cc7d74b1497de9af7ad7ccdd0
W20251029 08:52:30.279577 861 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0ab56045bd814d59831a98618c759af4
I20251029 08:52:30.280158 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_87f770c9be434c09a9ddef0f8768c953" } new_table_name: "default.Table_eb14554adb22488f885053e71d883ad5" modify_external_catalogs: true
I20251029 08:52:30.281430 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1091) took 2981 ms. Trace:
I20251029 08:52:30.281515 1179 rpcz_store.cc:276] 1029 08:52:27.299896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.299961 (+ 65us) service_pool.cc:225] Handling call
1029 08:52:30.281303 (+2981342us) ts_tablet_manager.cc:1151] Deleting tablet cea6d85081f246788ba29bb4a9c9a4e8
1029 08:52:30.281313 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.281410 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977415}
I20251029 08:52:30.281790 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 09968d93203140fa9201928c87702ee0 (table default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]) successfully deleted
I20251029 08:52:30.281874 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1092) took 2981 ms. Trace:
W20251029 08:52:30.281997 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f744556dccf348fd8b92071f224730aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f744556dccf348fd8b92071f224730aa already in progress: creating tablet
I20251029 08:52:30.281948 1179 rpcz_store.cc:276] 1029 08:52:27.299995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.300151 (+ 156us) service_pool.cc:225] Handling call
1029 08:52:30.281757 (+2981606us) ts_tablet_manager.cc:1151] Deleting tablet 38032950744b4426927157fad0962dfb
1029 08:52:30.281766 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.281858 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977804}
I20251029 08:52:30.282229 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1093) took 2982 ms. Trace:
I20251029 08:52:30.282292 1179 rpcz_store.cc:276] 1029 08:52:27.300130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.300176 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:30.282138 (+2981962us) ts_tablet_manager.cc:1151] Deleting tablet 46bcc468ce964610a857bc2b11b5b3e6
1029 08:52:30.282146 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.282216 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2974064}
I20251029 08:52:30.282613 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1098) took 2791 ms. Trace:
I20251029 08:52:30.282718 1179 rpcz_store.cc:276] 1029 08:52:27.490915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.490960 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:30.282525 (+2791565us) ts_tablet_manager.cc:1151] Deleting tablet 960f17ab6bed4ef5ad382421d84bf8d7
1029 08:52:30.282533 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.282599 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791408}
I20251029 08:52:30.282981 1179 tablet_replica.cc:333] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.283928 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.284396 1179 raft_consensus.cc:2243] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.284632 1179 raft_consensus.cc:2272] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.285248 1179 ts_tablet_manager.cc:1916] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.285756 1974 ts_tablet_manager.cc:1929] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.285887 1974 log.cc:1199] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/a8c4d57a59684723bfd27c7dda302ef3
I20251029 08:52:30.286868 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ab56045bd814d59831a98618c759af4" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:30.287074 1068 raft_consensus.cc:2468] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 0.
W20251029 08:52:30.287698 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cea6d85081f246788ba29bb4a9c9a4e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea6d85081f246788ba29bb4a9c9a4e8 already in progress: creating tablet
W20251029 08:52:30.287961 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 38032950744b4426927157fad0962dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38032950744b4426927157fad0962dfb already in progress: creating tablet
W20251029 08:52:30.288172 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 46bcc468ce964610a857bc2b11b5b3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bcc468ce964610a857bc2b11b5b3e6 already in progress: creating tablet
W20251029 08:52:30.288394 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 960f17ab6bed4ef5ad382421d84bf8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960f17ab6bed4ef5ad382421d84bf8d7 already in progress: creating tablet
I20251029 08:52:30.289523 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.291666 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:30.294807 998 consensus_peers.cc:597] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0fa089fce564af4b03aa323cec6cbf6. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.298400 1179 ts_tablet_manager.cc:1929] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.300499 764 tablet_service.cc:1505] Processing CreateTablet for tablet 307262da8888461ebddaaf33a31be8b8 (DEFAULT_TABLE table=default.Table_af67eaef7a72468c86b5c55ccb6d2626 [id=4feb43c217fa4d70823de1de3ccd3b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.301028 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 307262da8888461ebddaaf33a31be8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.301990 757 tablet_service.cc:1505] Processing CreateTablet for tablet a777b0d5fe6143818f7dba47ea86dcc0 (DEFAULT_TABLE table=default.Table_5a8ff23a00f145d4943422ba0649f47e [id=a5aff84c1dd74f39b30a589e998f9172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.302309 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777b0d5fe6143818f7dba47ea86dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.304208 1176 ts_tablet_manager.cc:1929] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.304308 1176 log.cc:1199] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/fcaa3fa779824b38887ae001623ba67b
I20251029 08:52:30.304657 1176 ts_tablet_manager.cc:1950] T fcaa3fa779824b38887ae001623ba67b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.305415 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1114) took 2614 ms. Trace:
I20251029 08:52:30.305512 1176 rpcz_store.cc:276] 1029 08:52:27.690997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.691049 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:30.275095 (+2584046us) ts_tablet_manager.cc:1151] Deleting tablet fcaa3fa779824b38887ae001623ba67b
1029 08:52:30.275106 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.287551 (+ 12445us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.304113 (+ 16562us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.305397 (+ 1284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2583700}
I20251029 08:52:30.305987 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet fcaa3fa779824b38887ae001623ba67b (table default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]) successfully deleted
I20251029 08:52:30.298976 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 213fc61402784f19a5287fdb57719616 (DEFAULT_TABLE table=default.Table_b17b82921a624ff5a4c04ddaa4462781 [id=32886fb555f5419b937be2cc966e82fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.313926 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213fc61402784f19a5287fdb57719616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.300581 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 307262da8888461ebddaaf33a31be8b8 (DEFAULT_TABLE table=default.Table_af67eaef7a72468c86b5c55ccb6d2626 [id=4feb43c217fa4d70823de1de3ccd3b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.317835 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 307262da8888461ebddaaf33a31be8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.313004 1183 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.318518 1183 ts_tablet_manager.cc:1403] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.001s
I20251029 08:52:30.319496 1183 raft_consensus.cc:359] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.319720 1183 raft_consensus.cc:385] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.320040 1183 raft_consensus.cc:740] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.320325 1183 consensus_queue.cc:260] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:30.322296 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 194726us
I20251029 08:52:30.327234 1179 log.cc:1199] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:30.327422 1178 tablet_bootstrap.cc:700] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent opening tablet: real 0.234s user 0.001s sys 0.000s
I20251029 08:52:30.327756 1178 tablet_bootstrap.cc:654] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.327996 1179 ts_tablet_manager.cc:1950] T 254e62d88c3242a0a3ac7470a8005d74 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.329224 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1099) took 2837 ms. Trace:
I20251029 08:52:30.329375 1179 rpcz_store.cc:276] 1029 08:52:27.491276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.491318 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:30.282939 (+2791621us) ts_tablet_manager.cc:1151] Deleting tablet 254e62d88c3242a0a3ac7470a8005d74
1029 08:52:30.282947 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.295157 (+ 12210us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.298284 (+ 3127us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.329202 (+ 30918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791473}
I20251029 08:52:30.329852 1179 tablet_replica.cc:333] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.330215 1179 raft_consensus.cc:2243] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.329542 2707 raft_consensus.cc:493] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.330495 1179 raft_consensus.cc:2272] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.330516 2707 raft_consensus.cc:515] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.331041 1179 ts_tablet_manager.cc:1916] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.331171 2707 leader_election.cc:290] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.332263 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf9e36c47be4509a998be8e7138475b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.332311 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1121) took 2483 ms. Trace:
I20251029 08:52:30.332399 1176 rpcz_store.cc:276] 1029 08:52:27.848431 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.848481 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:30.332149 (+2483668us) ts_tablet_manager.cc:1151] Deleting tablet 980e7125f59140c8876e4befc1becd78
1029 08:52:30.332162 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.332287 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2480246}
I20251029 08:52:30.332691 1176 tablet_replica.cc:333] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.332866 1176 raft_consensus.cc:2243] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.333009 1176 raft_consensus.cc:2272] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.333559 1176 ts_tablet_manager.cc:1916] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:30.333797 996 leader_election.cc:343] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:30.334323 998 leader_election.cc:343] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: dbf9e36c47be4509a998be8e7138475b
I20251029 08:52:30.334425 998 leader_election.cc:304] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.334820 2797 raft_consensus.cc:493] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.334945 2797 raft_consensus.cc:515] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.335548 2797 leader_election.cc:290] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.335680 2797 raft_consensus.cc:2749] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.336635 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:30.336843 773 raft_consensus.cc:2468] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:30.338167 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:30.338798 996 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.339905 998 leader_election.cc:304] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.340377 2707 raft_consensus.cc:2804] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:30.340453 2707 raft_consensus.cc:493] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:30.340521 2707 raft_consensus.cc:3060] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.342119 1179 ts_tablet_manager.cc:1929] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.342231 1179 log.cc:1199] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:30.342527 1179 ts_tablet_manager.cc:1950] T bf29e030892d44e1ad4d1ca4f5aeab3b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.342443 886 tablet_service.cc:1505] Processing CreateTablet for tablet 307262da8888461ebddaaf33a31be8b8 (DEFAULT_TABLE table=default.Table_af67eaef7a72468c86b5c55ccb6d2626 [id=4feb43c217fa4d70823de1de3ccd3b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.342862 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 307262da8888461ebddaaf33a31be8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.343209 2707 raft_consensus.cc:515] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.343323 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1100) took 2851 ms. Trace:
I20251029 08:52:30.343403 1179 rpcz_store.cc:276] 1029 08:52:27.491653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.491711 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:30.329801 (+2838090us) ts_tablet_manager.cc:1151] Deleting tablet bf29e030892d44e1ad4d1ca4f5aeab3b
1029 08:52:30.329811 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.334196 (+ 4385us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.341996 (+ 7800us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.343300 (+ 1304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837956}
I20251029 08:52:30.343724 1179 tablet_replica.cc:333] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.345966 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:30.346859 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "442c37e1ee724b6b96f679f05640baf9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:30.347070 773 raft_consensus.cc:3060] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.347339 1179 raft_consensus.cc:2243] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.347777 904 tablet_service.cc:1505] Processing CreateTablet for tablet a777b0d5fe6143818f7dba47ea86dcc0 (DEFAULT_TABLE table=default.Table_5a8ff23a00f145d4943422ba0649f47e [id=a5aff84c1dd74f39b30a589e998f9172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.348163 1179 raft_consensus.cc:2272] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.348208 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777b0d5fe6143818f7dba47ea86dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:30.348471 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 980e7125f59140c8876e4befc1becd78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 980e7125f59140c8876e4befc1becd78 already in progress: creating tablet
I20251029 08:52:30.348742 1179 ts_tablet_manager.cc:1916] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.348891 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 254e62d88c3242a0a3ac7470a8005d74 (table default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]) successfully deleted
I20251029 08:52:30.349062 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet bf29e030892d44e1ad4d1ca4f5aeab3b (table default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]) successfully deleted
W20251029 08:52:30.349371 996 leader_election.cc:343] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.350200 2707 leader_election.cc:290] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.353497 773 raft_consensus.cc:2468] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:30.354326 998 leader_election.cc:304] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.355013 2797 raft_consensus.cc:2804] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:30.355144 2797 raft_consensus.cc:697] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:30.355391 2797 consensus_queue.cc:237] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.358472 1177 tablet_bootstrap.cc:492] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:30.358724 1177 ts_tablet_manager.cc:1403] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.006s
I20251029 08:52:30.359601 1183 ts_tablet_manager.cc:1434] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251029 08:52:30.359958 1183 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.359836 1177 raft_consensus.cc:359] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.360178 1177 raft_consensus.cc:385] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.360285 1177 raft_consensus.cc:740] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.360671 1177 consensus_queue.cc:260] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.361071 1176 ts_tablet_manager.cc:1929] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.361199 1176 log.cc:1199] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b8f2b34642704cd6bb966b6dcd494d51
I20251029 08:52:30.361532 1176 ts_tablet_manager.cc:1950] T b8f2b34642704cd6bb966b6dcd494d51 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.361502 1177 ts_tablet_manager.cc:1434] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 08:52:30.361905 1177 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:30.362313 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1127) took 2297 ms. Trace:
I20251029 08:52:30.362417 1176 rpcz_store.cc:276] 1029 08:52:28.064471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.065347 (+ 876us) service_pool.cc:225] Handling call
1029 08:52:30.332647 (+2267300us) ts_tablet_manager.cc:1151] Deleting tablet b8f2b34642704cd6bb966b6dcd494d51
1029 08:52:30.332656 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.349647 (+ 16991us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.360949 (+ 11302us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.362288 (+ 1339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267124}
I20251029 08:52:30.362933 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b8f2b34642704cd6bb966b6dcd494d51 (table default.Table_d8be37074ea440ef9fbf1e9d596dbf39 [id=acab1181d22d454abea6817e3481f112]) successfully deleted
I20251029 08:52:30.363935 1974 ts_tablet_manager.cc:1950] T a8c4d57a59684723bfd27c7dda302ef3 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:30.364406 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1034) took 4692 ms. Trace:
I20251029 08:52:30.364550 1974 rpcz_store.cc:276] 1029 08:52:25.671731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.671779 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:29.830321 (+4158542us) ts_tablet_manager.cc:1151] Deleting tablet a8c4d57a59684723bfd27c7dda302ef3
1029 08:52:29.830332 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.230108 (+399776us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.285621 (+ 55513us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.364388 (+ 78767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4158429}
I20251029 08:52:30.365054 1183 tablet_bootstrap.cc:654] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.365208 1179 ts_tablet_manager.cc:1929] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.365373 1179 log.cc:1199] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:30.365491 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet a8c4d57a59684723bfd27c7dda302ef3 (table default.Table_0cde7b2972814447a6da2ec23d3b645f [id=e8998849b757469db422e816c4ae3f9b]) successfully deleted
I20251029 08:52:30.365749 1179 ts_tablet_manager.cc:1950] T 3e65b3b1c8d14743aca1465d95ace95f P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.365883 1974 tablet_replica.cc:333] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:30.366566 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1101) took 2874 ms. Trace:
I20251029 08:52:30.369791 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1134) took 2165 ms. Trace:
I20251029 08:52:30.369954 1176 rpcz_store.cc:276] 1029 08:52:28.204132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.204202 (+ 70us) service_pool.cc:225] Handling call
1029 08:52:30.369628 (+2165426us) ts_tablet_manager.cc:1151] Deleting tablet 56f9308235f145ee838a55434f772bfa
1029 08:52:30.369641 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.369765 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165125}
I20251029 08:52:30.370030 1179 rpcz_store.cc:276] 1029 08:52:27.492001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.492060 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:30.343661 (+2851601us) ts_tablet_manager.cc:1151] Deleting tablet 3e65b3b1c8d14743aca1465d95ace95f
1029 08:52:30.343673 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.353493 (+ 9820us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.364965 (+ 11472us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.366546 (+ 1581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2851471}
W20251029 08:52:30.370468 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 56f9308235f145ee838a55434f772bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f9308235f145ee838a55434f772bfa already in progress: creating tablet
I20251029 08:52:30.370538 1179 tablet_replica.cc:333] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.370723 1179 raft_consensus.cc:2243] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.370729 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3e65b3b1c8d14743aca1465d95ace95f (table default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]) successfully deleted
I20251029 08:52:30.370975 1179 raft_consensus.cc:2272] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.371145 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1135) took 2127 ms. Trace:
I20251029 08:52:30.371230 1176 rpcz_store.cc:276] 1029 08:52:28.243688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.243761 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:30.371014 (+2127253us) ts_tablet_manager.cc:1151] Deleting tablet bdfda0866a614165824d1a3f7ad89af7
1029 08:52:30.371025 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.371125 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2127039}
W20251029 08:52:30.371644 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet bdfda0866a614165824d1a3f7ad89af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdfda0866a614165824d1a3f7ad89af7 already in progress: creating tablet
I20251029 08:52:30.371766 1179 ts_tablet_manager.cc:1916] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.372076 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1136) took 2125 ms. Trace:
I20251029 08:52:30.372200 1176 rpcz_store.cc:276] 1029 08:52:28.246576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.246640 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:30.371944 (+2125304us) ts_tablet_manager.cc:1151] Deleting tablet 420c3357ebff4dd08df4ed2477b40ddf
1029 08:52:30.371954 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.372059 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125084}
W20251029 08:52:30.372637 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 420c3357ebff4dd08df4ed2477b40ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420c3357ebff4dd08df4ed2477b40ddf already in progress: creating tablet
I20251029 08:52:30.372996 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1137) took 2097 ms. Trace:
I20251029 08:52:30.373100 1176 rpcz_store.cc:276] 1029 08:52:28.275640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.275720 (+ 80us) service_pool.cc:225] Handling call
1029 08:52:30.372899 (+2097179us) ts_tablet_manager.cc:1151] Deleting tablet c5d9527d294c432db2ccf60af84731b0
1029 08:52:30.372909 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.372981 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096947}
W20251029 08:52:30.373553 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c5d9527d294c432db2ccf60af84731b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d9527d294c432db2ccf60af84731b0 already in progress: creating tablet
I20251029 08:52:30.373912 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1138) took 2090 ms. Trace:
I20251029 08:52:30.374022 1176 rpcz_store.cc:276] 1029 08:52:28.283208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.283274 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:30.373812 (+2090538us) ts_tablet_manager.cc:1151] Deleting tablet 5e35d40b730a4a7883c324a6cc93b161
1029 08:52:30.373822 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.373896 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090322}
W20251029 08:52:30.374411 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5e35d40b730a4a7883c324a6cc93b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e35d40b730a4a7883c324a6cc93b161 already in progress: creating tablet
I20251029 08:52:30.374728 1176 tablet_replica.cc:333] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.374985 1176 raft_consensus.cc:2243] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.377638 1176 raft_consensus.cc:2272] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.378295 1179 ts_tablet_manager.cc:1929] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.380990 1176 ts_tablet_manager.cc:1916] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.387418 1176 ts_tablet_manager.cc:1929] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.387547 1176 log.cc:1199] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/7046bb1fcdd34d308aecddfdf0752911
I20251029 08:52:30.387691 1179 log.cc:1199] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:30.387871 1176 ts_tablet_manager.cc:1950] T 7046bb1fcdd34d308aecddfdf0752911 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.382360 1183 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.388201 1183 ts_tablet_manager.cc:1403] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20251029 08:52:30.388226 1179 ts_tablet_manager.cc:1950] T c9bbf6401423411089c9070f3b03f1a7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.389019 1183 raft_consensus.cc:359] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.390972 1183 raft_consensus.cc:385] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.391069 1183 raft_consensus.cc:740] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.391114 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1102) took 2897 ms. Trace:
I20251029 08:52:30.391300 1179 rpcz_store.cc:276] 1029 08:52:27.493314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.493356 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:30.370482 (+2877126us) ts_tablet_manager.cc:1151] Deleting tablet c9bbf6401423411089c9070f3b03f1a7
1029 08:52:30.370496 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.375055 (+ 4559us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.378188 (+ 3133us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.391033 (+ 12845us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1838,"tablet-delete.queue_time_us":2876973}
I20251029 08:52:30.391907 1179 tablet_replica.cc:333] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.392122 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet c9bbf6401423411089c9070f3b03f1a7 (table default.Table_18c407e625464254aa01f6cc9e12144f [id=58aabf0c50d64caa918b4d97574d7c5f]) successfully deleted
I20251029 08:52:30.392186 1179 raft_consensus.cc:2243] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:30.392531 1179 raft_consensus.cc:2272] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.391309 1183 consensus_queue.cc:260] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.393487 1179 ts_tablet_manager.cc:1916] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.395234 1183 ts_tablet_manager.cc:1434] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:30.395558 1183 tablet_bootstrap.cc:492] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.396013 886 tablet_service.cc:1505] Processing CreateTablet for tablet f70c026d5a194c808d2f183146ebadb4 (DEFAULT_TABLE table=default.Table_3c395564e2fb41c6b92c0abda8d3c16f [id=4a671713bb5d4c269612a46130b848e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.396588 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c026d5a194c808d2f183146ebadb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.397096 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1148) took 1886 ms. Trace:
I20251029 08:52:30.397192 1176 rpcz_store.cc:276] 1029 08:52:28.510588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.510710 (+ 122us) service_pool.cc:225] Handling call
1029 08:52:30.374683 (+1863973us) ts_tablet_manager.cc:1151] Deleting tablet 7046bb1fcdd34d308aecddfdf0752911
1029 08:52:30.374692 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.384274 (+ 9582us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.387324 (+ 3050us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.397073 (+ 9749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863738}
I20251029 08:52:30.397620 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 7046bb1fcdd34d308aecddfdf0752911 (table default.Table_e9b105d36aed4a0da8586038f56dec12 [id=463d151caf764effa17b75b7875c5e6f]) successfully deleted
I20251029 08:52:30.397991 1176 tablet_replica.cc:333] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.398206 1176 raft_consensus.cc:2243] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.401609 1047 tablet_service.cc:1505] Processing CreateTablet for tablet a777b0d5fe6143818f7dba47ea86dcc0 (DEFAULT_TABLE table=default.Table_5a8ff23a00f145d4943422ba0649f47e [id=a5aff84c1dd74f39b30a589e998f9172]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.402092 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777b0d5fe6143818f7dba47ea86dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.404276 1183 tablet_bootstrap.cc:654] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.406023 1179 ts_tablet_manager.cc:1929] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.407068 1176 raft_consensus.cc:2272] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.407254 1179 log.cc:1199] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:30.407662 1179 ts_tablet_manager.cc:1950] T d50e2fdfc0e64516a41c90f5a56f4c89 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.407692 1176 ts_tablet_manager.cc:1916] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.411118 2295 consensus_queue.cc:1048] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:30.412138 861 leader_election.cc:304] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:30.412254 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1103) took 2918 ms. Trace:
I20251029 08:52:30.412498 2295 raft_consensus.cc:2804] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:30.412577 2295 raft_consensus.cc:493] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:30.412400 1179 rpcz_store.cc:276] 1029 08:52:27.493647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.493696 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:30.391849 (+2898153us) ts_tablet_manager.cc:1151] Deleting tablet d50e2fdfc0e64516a41c90f5a56f4c89
1029 08:52:30.391860 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.398648 (+ 6788us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.405917 (+ 7269us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.412234 (+ 6317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2898025}
I20251029 08:52:30.412642 2295 raft_consensus.cc:3060] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.413033 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d50e2fdfc0e64516a41c90f5a56f4c89 (table default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]) successfully deleted
I20251029 08:52:30.413463 1179 tablet_replica.cc:333] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.413717 1179 raft_consensus.cc:2243] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.413935 1179 raft_consensus.cc:2272] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.414336 1176 ts_tablet_manager.cc:1929] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.414474 1176 log.cc:1199] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4e3a2fab467348099a4a1fec74eb5e65
I20251029 08:52:30.414824 1176 ts_tablet_manager.cc:1950] T 4e3a2fab467348099a4a1fec74eb5e65 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.415079 1179 ts_tablet_manager.cc:1916] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.415593 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1150) took 1904 ms. Trace:
I20251029 08:52:30.415683 1176 rpcz_store.cc:276] 1029 08:52:28.510851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.510993 (+ 142us) service_pool.cc:225] Handling call
1029 08:52:30.397936 (+1886943us) ts_tablet_manager.cc:1151] Deleting tablet 4e3a2fab467348099a4a1fec74eb5e65
1029 08:52:30.397949 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.410962 (+ 13013us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.414223 (+ 3261us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.415571 (+ 1348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886762}
I20251029 08:52:30.416139 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4e3a2fab467348099a4a1fec74eb5e65 (table default.Table_c68d96e4128e496097a5c5a4740db7ac [id=09dcc91cf1e045169ca0eedfe1e6dad3]) successfully deleted
I20251029 08:52:30.416239 1183 tablet_bootstrap.cc:492] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.416363 1183 ts_tablet_manager.cc:1403] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251029 08:52:30.416388 2295 raft_consensus.cc:515] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.417018 2295 leader_election.cc:290] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.417088 1183 raft_consensus.cc:359] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.417243 1183 raft_consensus.cc:385] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.417300 1183 raft_consensus.cc:740] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.417531 1974 raft_consensus.cc:2243] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:30.417567 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1147) took 1907 ms. Trace:
I20251029 08:52:30.417651 1176 rpcz_store.cc:276] 1029 08:52:28.510486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.520089 (+ 9603us) service_pool.cc:225] Handling call
1029 08:52:30.417420 (+1897331us) ts_tablet_manager.cc:1151] Deleting tablet e14d1d3421a542ec960d21453bb713ec
1029 08:52:30.417431 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.417549 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1897133}
I20251029 08:52:30.417711 2855 tablet.cc:1722] T 85b300f17a704f7dba1aa3724fcc2de7 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 08:52:30.418037 1974 raft_consensus.cc:2272] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:30.418087 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e14d1d3421a542ec960d21453bb713ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d1d3421a542ec960d21453bb713ec already in progress: creating tablet
I20251029 08:52:30.418632 1974 ts_tablet_manager.cc:1916] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.420698 1176 tablet_replica.cc:333] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.420930 1176 raft_consensus.cc:2243] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.421082 1176 raft_consensus.cc:2272] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.421590 1176 ts_tablet_manager.cc:1916] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.422174 764 tablet_service.cc:1505] Processing CreateTablet for tablet f70c026d5a194c808d2f183146ebadb4 (DEFAULT_TABLE table=default.Table_3c395564e2fb41c6b92c0abda8d3c16f [id=4a671713bb5d4c269612a46130b848e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.422541 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c026d5a194c808d2f183146ebadb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.422497 1183 consensus_queue.cc:260] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.423180 1046 tablet_service.cc:1505] Processing CreateTablet for tablet f70c026d5a194c808d2f183146ebadb4 (DEFAULT_TABLE table=default.Table_3c395564e2fb41c6b92c0abda8d3c16f [id=4a671713bb5d4c269612a46130b848e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.423583 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c026d5a194c808d2f183146ebadb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.423832 904 tablet_service.cc:1505] Processing CreateTablet for tablet 412c2b253aab48aeb2ce1f8c4c725113 (DEFAULT_TABLE table=default.Table_4dfe9cad106348adb513ca14b0bd54bb [id=4542d509cec24e8d9aebef0bb50cf5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.424218 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412c2b253aab48aeb2ce1f8c4c725113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.427809 1176 ts_tablet_manager.cc:1929] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.427920 1176 log.cc:1199] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/663d6587a5e14519afbb0f94d691d1b4
I20251029 08:52:30.430424 1183 ts_tablet_manager.cc:1434] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 08:52:30.431370 1178 tablet_bootstrap.cc:492] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:30.431572 1178 ts_tablet_manager.cc:1403] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.340s user 0.002s sys 0.002s
I20251029 08:52:30.433178 1177 tablet_bootstrap.cc:654] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.434998 1176 ts_tablet_manager.cc:1950] T 663d6587a5e14519afbb0f94d691d1b4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.436429 1183 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.439160 1179 ts_tablet_manager.cc:1929] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.439383 1179 log.cc:1199] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:30.439854 1179 ts_tablet_manager.cc:1950] T d3955ff79faf42bb82f764aae79b85d9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.440868 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1104) took 2946 ms. Trace:
I20251029 08:52:30.441020 1179 rpcz_store.cc:276] 1029 08:52:27.493989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.494027 (+ 38us) service_pool.cc:225] Handling call
1029 08:52:30.413384 (+2919357us) ts_tablet_manager.cc:1151] Deleting tablet d3955ff79faf42bb82f764aae79b85d9
1029 08:52:30.413393 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.418357 (+ 4964us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.439032 (+ 20675us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.440848 (+ 1816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2919232}
I20251029 08:52:30.441377 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ab56045bd814d59831a98618c759af4" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:30.441565 1067 raft_consensus.cc:3060] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.441668 1183 tablet_bootstrap.cc:654] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.441608 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d3955ff79faf42bb82f764aae79b85d9 (table default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]) successfully deleted
I20251029 08:52:30.441998 1179 tablet_replica.cc:333] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.442194 1179 raft_consensus.cc:2243] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.442348 1179 raft_consensus.cc:2272] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.442903 1179 ts_tablet_manager.cc:1916] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.444806 1067 raft_consensus.cc:2468] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927689e0dc5e4bc79d4a9d79a873f088 in term 1.
I20251029 08:52:30.445791 1178 raft_consensus.cc:359] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.445977 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1149) took 1935 ms. Trace:
I20251029 08:52:30.446135 1178 raft_consensus.cc:385] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.446233 1178 raft_consensus.cc:740] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.446120 1176 rpcz_store.cc:276] 1029 08:52:28.510732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.520321 (+ 9589us) service_pool.cc:225] Handling call
1029 08:52:30.420636 (+1900315us) ts_tablet_manager.cc:1151] Deleting tablet 663d6587a5e14519afbb0f94d691d1b4
1029 08:52:30.420651 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.424743 (+ 4092us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.427721 (+ 2978us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.445952 (+ 18231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10120,"tablet-delete.queue_time_us":1900195}
I20251029 08:52:30.446664 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1163) took 1333 ms. Trace:
I20251029 08:52:30.446637 1178 consensus_queue.cc:260] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.447157 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 663d6587a5e14519afbb0f94d691d1b4 (table default.Table_ab4e50ed7bad43b4adab2ec2cf76a327 [id=cf151f9ad5974db78cd578b1edfd034b]) successfully deleted
I20251029 08:52:30.447276 1178 ts_tablet_manager.cc:1434] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251029 08:52:30.446748 1176 rpcz_store.cc:276] 1029 08:52:29.113310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.140600 (+ 27290us) service_pool.cc:225] Handling call
1029 08:52:30.446521 (+1305921us) ts_tablet_manager.cc:1151] Deleting tablet d1d2df9e1e744ed1a6cf549ab57ed266
1029 08:52:30.446534 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.446647 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304583}
W20251029 08:52:30.447857 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d1d2df9e1e744ed1a6cf549ab57ed266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d2df9e1e744ed1a6cf549ab57ed266 already in progress: creating tablet
I20251029 08:52:30.450775 764 tablet_service.cc:1505] Processing CreateTablet for tablet 412c2b253aab48aeb2ce1f8c4c725113 (DEFAULT_TABLE table=default.Table_4dfe9cad106348adb513ca14b0bd54bb [id=4542d509cec24e8d9aebef0bb50cf5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.451609 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 412c2b253aab48aeb2ce1f8c4c725113 (DEFAULT_TABLE table=default.Table_4dfe9cad106348adb513ca14b0bd54bb [id=4542d509cec24e8d9aebef0bb50cf5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.452127 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412c2b253aab48aeb2ce1f8c4c725113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.443630 2855 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:30.462922 1183 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.463187 1183 ts_tablet_manager.cc:1403] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20251029 08:52:30.464216 1183 raft_consensus.cc:359] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.464520 1183 raft_consensus.cc:385] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.464636 1183 raft_consensus.cc:740] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.464954 1183 consensus_queue.cc:260] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.465971 1183 ts_tablet_manager.cc:1434] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 08:52:30.466547 1183 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.472465 1179 ts_tablet_manager.cc:1929] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.472733 1179 log.cc:1199] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e2b87d133f8a4bd1877e07a1a8758147
I20251029 08:52:30.473145 1179 ts_tablet_manager.cc:1950] T e2b87d133f8a4bd1877e07a1a8758147 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.474086 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1111) took 2657 ms. Trace:
I20251029 08:52:30.474244 1179 rpcz_store.cc:276] 1029 08:52:27.816705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.817537 (+ 832us) service_pool.cc:225] Handling call
1029 08:52:30.441946 (+2624409us) ts_tablet_manager.cc:1151] Deleting tablet e2b87d133f8a4bd1877e07a1a8758147
1029 08:52:30.441956 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.446000 (+ 4044us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.472329 (+ 26329us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.474060 (+ 1731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2624225}
I20251029 08:52:30.474943 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e2b87d133f8a4bd1877e07a1a8758147 (table default.Table_b7cdbb09ce36440b99a8c95ab0f3c0da [id=85dda4d8658f4607bafddcfba7b8678a]) successfully deleted
I20251029 08:52:30.475335 1183 tablet_bootstrap.cc:654] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.475567 1179 tablet_replica.cc:333] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.475960 1179 raft_consensus.cc:2243] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.476214 1179 raft_consensus.cc:2272] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.476821 1179 ts_tablet_manager.cc:1916] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.488445 1183 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.488631 1183 ts_tablet_manager.cc:1403] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251029 08:52:30.489439 1183 raft_consensus.cc:359] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.489646 1183 raft_consensus.cc:385] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.489706 1183 raft_consensus.cc:740] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.489950 1183 consensus_queue.cc:260] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.492130 1179 ts_tablet_manager.cc:1929] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.492324 1179 log.cc:1199] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2fb380018f46465e82ac6223ecf8dd0d
I20251029 08:52:30.492416 1183 ts_tablet_manager.cc:1434] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:30.492746 1183 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.492895 1179 ts_tablet_manager.cc:1950] T 2fb380018f46465e82ac6223ecf8dd0d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.494160 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1112) took 2675 ms. Trace:
I20251029 08:52:30.501715 1179 rpcz_store.cc:276] 1029 08:52:27.818260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.825709 (+ 7449us) service_pool.cc:225] Handling call
1029 08:52:30.475508 (+2649799us) ts_tablet_manager.cc:1151] Deleting tablet 2fb380018f46465e82ac6223ecf8dd0d
1029 08:52:30.475520 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.480262 (+ 4742us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.489590 (+ 9328us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.494139 (+ 4549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2648668}
I20251029 08:52:30.502293 1179 tablet_replica.cc:333] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.502578 1179 raft_consensus.cc:2243] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.502861 1179 raft_consensus.cc:2272] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.494516 1183 tablet_bootstrap.cc:654] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.503629 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2fb380018f46465e82ac6223ecf8dd0d (table default.Table_1d27485babbd4ce5afd764a5d187d095 [id=ccc56f591675423da2206c023b1f8853]) successfully deleted
I20251029 08:52:30.504947 1179 ts_tablet_manager.cc:1916] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:30.505856 861 consensus_peers.cc:597] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e65b3b1c8d14743aca1465d95ace95f. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.506098 861 leader_election.cc:304] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088; no voters:
I20251029 08:52:30.506503 2731 raft_consensus.cc:2804] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:30.506788 2731 raft_consensus.cc:697] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Becoming Leader. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Running, Role: LEADER
I20251029 08:52:30.507082 2731 consensus_queue.cc:237] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.510052 757 tablet_service.cc:1505] Processing CreateTablet for tablet f73a300a67f64cd6b6e9e9bb2e82db69 (DEFAULT_TABLE table=default.Table_2d36524247274c099382f1d22aa6dd76 [id=1e5fb3a6a5ba40b0b192106bca8376bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.510560 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_44fb5557e46d4baf91ba74fd423b6604"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.510804 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 31bed64a27164baa8177b87b54b5947b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e3782c226c14143aec56367d1a04996) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:30.511166 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44fb5557e46d4baf91ba74fd423b6604 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.511883 590 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_3584ce3292564e359247b74ca750e4d9 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 1 got 0
I20251029 08:52:30.512523 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0fb0c73465d84a63805ee4f2b426b6de" } new_table_name: "default.Table_42bb7e9fea2f454681790db1b78b0683" modify_external_catalogs: true
I20251029 08:52:30.512852 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3efff2709af84a29acfd361aa9213152" } new_table_name: "default.Table_cfcd92d4157142e99ededb7b68d63193" modify_external_catalogs: true
I20251029 08:52:30.514083 1178 tablet_bootstrap.cc:492] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:30.514029 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7ee2369d41744dd6a935286ad2dfdfdf" } modify_external_catalogs: true
I20251029 08:52:30.514079 594 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3bd45ebf7bd64e5cbb19662e966bb925" } new_table_name: "default.Table_538d99e1eab6407ea3f0899f8db94c96" modify_external_catalogs: true
I20251029 08:52:30.514158 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_7ee2369d41744dd6a935286ad2dfdfdf" } modify_external_catalogs: true
I20251029 08:52:30.515620 1178 tablet_bootstrap.cc:654] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.515558 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b68078657014c3790ceed68233e826e" } modify_external_catalogs: true
I20251029 08:52:30.515734 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4b68078657014c3790ceed68233e826e" } modify_external_catalogs: true
I20251029 08:52:30.517333 891 tablet_service.cc:1505] Processing CreateTablet for tablet f73a300a67f64cd6b6e9e9bb2e82db69 (DEFAULT_TABLE table=default.Table_2d36524247274c099382f1d22aa6dd76 [id=1e5fb3a6a5ba40b0b192106bca8376bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.517709 2797 raft_consensus.cc:493] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.517856 2797 raft_consensus.cc:515] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.518285 1046 tablet_service.cc:1505] Processing CreateTablet for tablet f73a300a67f64cd6b6e9e9bb2e82db69 (DEFAULT_TABLE table=default.Table_2d36524247274c099382f1d22aa6dd76 [id=1e5fb3a6a5ba40b0b192106bca8376bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.518513 2797 leader_election.cc:290] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.518695 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73a300a67f64cd6b6e9e9bb2e82db69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.520610 2295 raft_consensus.cc:493] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.520797 2295 raft_consensus.cc:515] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.521529 1179 ts_tablet_manager.cc:1929] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.521641 1179 log.cc:1199] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7a355c68ecfb45ff8c396873d91268a8
I20251029 08:52:30.521826 2295 leader_election.cc:290] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.521961 1179 ts_tablet_manager.cc:1950] T 7a355c68ecfb45ff8c396873d91268a8 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.522774 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1113) took 2691 ms. Trace:
I20251029 08:52:30.524847 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_66cf9e18bddd418e88537d60dbac69ba" } new_table_name: "default.Table_6090d35adbbd4757b87805278db68aa5" modify_external_catalogs: true
W20251029 08:52:30.524701 998 consensus_peers.cc:597] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcaa3fa779824b38887ae001623ba67b. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.527012 1179 rpcz_store.cc:276] 1029 08:52:27.830887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.830951 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:30.502235 (+2671284us) ts_tablet_manager.cc:1151] Deleting tablet 7a355c68ecfb45ff8c396873d91268a8
1029 08:52:30.502248 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.508614 (+ 6366us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.521407 (+ 12793us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.522752 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2671100}
I20251029 08:52:30.527630 1179 tablet_replica.cc:333] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.528321 1179 raft_consensus.cc:2243] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.529600 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.530880 1179 raft_consensus.cc:2272] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.531688 1179 ts_tablet_manager.cc:1916] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.531874 2786 raft_consensus.cc:493] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.532104 2786 raft_consensus.cc:515] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.532799 2786 leader_election.cc:290] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:30.533247 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d26ff78aa5f94ae49a197daf2be00aa5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:30.533792 720 leader_election.cc:343] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:30.532716 861 leader_election.cc:343] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0ab56045bd814d59831a98618c759af4
I20251029 08:52:30.534155 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d26ff78aa5f94ae49a197daf2be00aa5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.535900 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7a355c68ecfb45ff8c396873d91268a8 (table default.Table_a3dc21093f304d13933e68c784dbfe5e [id=91274c3e8fac4bdbbbffefa10cd73044]) successfully deleted
W20251029 08:52:30.536363 718 leader_election.cc:343] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.536468 718 leader_election.cc:304] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.537047 2786 raft_consensus.cc:2749] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.537869 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73a300a67f64cd6b6e9e9bb2e82db69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.538060 901 tablet_service.cc:1505] Processing CreateTablet for tablet b48fda3b7ac846dd8c05ced4eee5bd4e (DEFAULT_TABLE table=default.Table_da1f7a7ff5004099ab9ac18ce51a23d5 [id=311e4fc75c124beda85f6946144c36b0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:30.539820 996 consensus_peers.cc:597] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.539865 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:30.540395 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412c2b253aab48aeb2ce1f8c4c725113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.542212 2786 raft_consensus.cc:493] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.542361 2786 raft_consensus.cc:515] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.542958 2786 leader_election.cc:290] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.543820 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31bed64a27164baa8177b87b54b5947b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:30.544176 773 raft_consensus.cc:2468] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:30.544453 720 leader_election.cc:343] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.544890 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1164) took 1331 ms. Trace:
W20251029 08:52:30.537125 861 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:30.544994 1176 rpcz_store.cc:276] 1029 08:52:29.213447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.213622 (+ 175us) service_pool.cc:225] Handling call
1029 08:52:30.447561 (+1233939us) ts_tablet_manager.cc:1151] Deleting tablet c9b611ae08a247d3ba02eb177c6b1b12
1029 08:52:30.544754 (+ 97193us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.544873 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233703}
I20251029 08:52:30.545826 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1165) took 1268 ms. Trace:
I20251029 08:52:30.545912 1176 rpcz_store.cc:276] 1029 08:52:29.277708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.277759 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:30.545289 (+1267530us) ts_tablet_manager.cc:1151] Deleting tablet b61829fcd59d4ad4ab427270b7089d57
1029 08:52:30.545692 (+ 403us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.545808 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267320}
I20251029 08:52:30.548936 886 tablet_service.cc:1505] Processing CreateTablet for tablet 69646d1c8eb843ec82f25cbd7aa65849 (DEFAULT_TABLE table=default.Table_b2c950d7747b4bfebfb385e8fb2f8d59 [id=9c9db0fbaf4940e0a56469c15a806c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.550338 1176 tablet_replica.cc:333] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:30.550550 1176 raft_consensus.cc:2243] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.550717 1176 raft_consensus.cc:2272] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.551309 1176 ts_tablet_manager.cc:1916] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:30.549903 861 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:30.551522 861 leader_election.cc:304] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:30.553071 1179 ts_tablet_manager.cc:1929] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 08:52:30.553845 996 consensus_peers.cc:597] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:30.554204 1179 log.cc:1199] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:30.554437 749 tablet_service.cc:1505] Processing CreateTablet for tablet b48fda3b7ac846dd8c05ced4eee5bd4e (DEFAULT_TABLE table=default.Table_da1f7a7ff5004099ab9ac18ce51a23d5 [id=311e4fc75c124beda85f6946144c36b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.554752 1179 ts_tablet_manager.cc:1950] T 913fb69271e14fe3a8d0b29d0db0dc8e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.554930 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fda3b7ac846dd8c05ced4eee5bd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.555552 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1119) took 2699 ms. Trace:
I20251029 08:52:30.555650 1179 rpcz_store.cc:276] 1029 08:52:27.856176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.856227 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:30.527557 (+2671330us) ts_tablet_manager.cc:1151] Deleting tablet 913fb69271e14fe3a8d0b29d0db0dc8e
1029 08:52:30.527579 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.542076 (+ 14497us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.552946 (+ 10870us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.555527 (+ 2581us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328,"tablet-delete.queue_time_us":2671155}
I20251029 08:52:30.556926 593 catalog_manager.cc:5649] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:30.557329 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 442c37e1ee724b6b96f679f05640baf9 (table default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb]). Expected version 1 got 0
I20251029 08:52:30.557535 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:30.557683 593 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:30.560227 2731 raft_consensus.cc:2749] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.560629 998 leader_election.cc:304] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:30.561092 2707 raft_consensus.cc:2804] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:30.561168 2707 raft_consensus.cc:493] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:30.561220 2707 raft_consensus.cc:3060] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:30.561594 1179 tablet_replica.cc:333] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.561802 1179 raft_consensus.cc:2243] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.561985 1179 raft_consensus.cc:2272] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.562582 1179 ts_tablet_manager.cc:1916] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.563308 2731 raft_consensus.cc:493] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.563445 2731 raft_consensus.cc:515] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.564108 2731 leader_election.cc:290] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.564317 2731 raft_consensus.cc:493] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.564455 2731 raft_consensus.cc:515] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.565124 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31bed64a27164baa8177b87b54b5947b" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.565208 2731 leader_election.cc:290] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.566898 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 913fb69271e14fe3a8d0b29d0db0dc8e (table default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]) successfully deleted
W20251029 08:52:30.567214 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c9b611ae08a247d3ba02eb177c6b1b12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b611ae08a247d3ba02eb177c6b1b12 already in progress: creating tablet
W20251029 08:52:30.567412 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b61829fcd59d4ad4ab427270b7089d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61829fcd59d4ad4ab427270b7089d57 already in progress: creating tablet
W20251029 08:52:30.567605 584 catalog_manager.cc:4952] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet b79ec90d93ee48abaaed3bc3cd41bf62 because the tablet was not found. No further retry: Not found: Tablet not found: b79ec90d93ee48abaaed3bc3cd41bf62
W20251029 08:52:30.567975 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 297945c50119432a9f9d3e356d573879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 297945c50119432a9f9d3e356d573879 already in progress: creating tablet
W20251029 08:52:30.568331 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet adba5919a81348bb94569ca52fa4bded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adba5919a81348bb94569ca52fa4bded already in progress: creating tablet
W20251029 08:52:30.568548 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5d292c29d2b34086a5976da251f9ed5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d292c29d2b34086a5976da251f9ed5e already in progress: creating tablet
W20251029 08:52:30.568751 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 76dcdd90e4a54cbfb0cb62192c465c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76dcdd90e4a54cbfb0cb62192c465c40 already in progress: creating tablet
W20251029 08:52:30.568934 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8017e8fe5f634ef0bf7f8be7c8aea033 already in progress: creating tablet
I20251029 08:52:30.569684 1051 tablet_service.cc:1505] Processing CreateTablet for tablet b48fda3b7ac846dd8c05ced4eee5bd4e (DEFAULT_TABLE table=default.Table_da1f7a7ff5004099ab9ac18ce51a23d5 [id=311e4fc75c124beda85f6946144c36b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.570179 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fda3b7ac846dd8c05ced4eee5bd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.571210 2797 raft_consensus.cc:493] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.571368 2797 raft_consensus.cc:515] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.572044 2797 leader_election.cc:290] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.572868 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eeabf8f23751447aa9af02363c40361d" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:30.573674 996 leader_election.cc:343] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.574278 2797 raft_consensus.cc:493] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.574424 2797 raft_consensus.cc:515] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:30.575484 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbfcb4e264cb401ab88ed9437775ad98 consensus state has no leader
I20251029 08:52:30.575848 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_01f74d8d149e47da972673a83c50e16c" } modify_external_catalogs: true
I20251029 08:52:30.576005 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_01f74d8d149e47da972673a83c50e16c" } modify_external_catalogs: true
I20251029 08:52:30.577559 595 master_service.cc:954] ReplaceTablet: received request to replace tablet 07bf5d693be240f18bacccc81a6ea5d4 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:30.579442 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b52314139945dd9c2e6ec899bf06dc" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.578811 2797 leader_election.cc:290] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:30.580825 996 leader_election.cc:343] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.589267 2302 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" instance_seqno: 1761727920815568) as {username='slave'} at 127.0.104.2:56167; Asking this server to re-register.
I20251029 08:52:30.592186 1179 ts_tablet_manager.cc:1929] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251029 08:52:30.592726 861 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 254e62d88c3242a0a3ac7470a8005d74
W20251029 08:52:30.592988 861 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 6689cf0804f946c18ebc555a191549d4
W20251029 08:52:30.593197 861 consensus_peers.cc:597] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e65b3b1c8d14743aca1465d95ace95f. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:30.594460 861 consensus_peers.cc:597] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e213713757949caa3f504f70026b7b5. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:30.594656 861 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:30.594792 861 leader_election.cc:304] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:30.595046 2731 raft_consensus.cc:2749] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.595196 1179 log.cc:1199] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0cc95257c937454aa6cd55b5a9394a56
I20251029 08:52:30.595857 1179 ts_tablet_manager.cc:1950] T 0cc95257c937454aa6cd55b5a9394a56 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.597395 972 heartbeater.cc:461] Registering TS with master...
I20251029 08:52:30.598009 1974 ts_tablet_manager.cc:1929] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.598186 1974 log.cc:1199] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/652903cbbdb042d780e7d71fc2c0ab4d
I20251029 08:52:30.598335 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1120) took 2741 ms. Trace:
I20251029 08:52:30.598644 1974 ts_tablet_manager.cc:1950] T 652903cbbdb042d780e7d71fc2c0ab4d P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:30.601851 718 leader_election.cc:343] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.602010 718 leader_election.cc:304] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.602089 1179 rpcz_store.cc:276] 1029 08:52:27.856529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.856595 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:30.561533 (+2704938us) ts_tablet_manager.cc:1151] Deleting tablet 0cc95257c937454aa6cd55b5a9394a56
1029 08:52:30.561545 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.565950 (+ 4405us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.592054 (+ 26104us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.598308 (+ 6254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704780}
I20251029 08:52:30.602255 2786 raft_consensus.cc:2749] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.602614 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]) successfully deleted
I20251029 08:52:30.563931 2707 raft_consensus.cc:515] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.603372 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1123) took 2721 ms. Trace:
I20251029 08:52:30.603981 1179 rpcz_store.cc:276] 1029 08:52:27.881399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.881446 (+ 47us) service_pool.cc:225] Handling call
1029 08:52:30.603227 (+2721781us) ts_tablet_manager.cc:1151] Deleting tablet 1fb9c8c7286149128c2741635c242341
1029 08:52:30.603238 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.603345 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2721577}
W20251029 08:52:30.604739 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 1fb9c8c7286149128c2741635c242341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb9c8c7286149128c2741635c242341 already in progress: creating tablet
I20251029 08:52:30.605528 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1128) took 2706 ms. Trace:
I20251029 08:52:30.605909 1179 rpcz_store.cc:276] 1029 08:52:27.898979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.899787 (+ 808us) service_pool.cc:225] Handling call
1029 08:52:30.605412 (+2705625us) ts_tablet_manager.cc:1151] Deleting tablet 980e7125f59140c8876e4befc1becd78
1029 08:52:30.605421 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.605510 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705459}
W20251029 08:52:30.606693 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 980e7125f59140c8876e4befc1becd78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 980e7125f59140c8876e4befc1becd78 already in progress: creating tablet
I20251029 08:52:30.607132 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1117) took 2757 ms. Trace:
I20251029 08:52:30.607492 1179 rpcz_store.cc:276] 1029 08:52:27.849247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.952188 (+102941us) service_pool.cc:225] Handling call
1029 08:52:30.607030 (+2654842us) ts_tablet_manager.cc:1151] Deleting tablet 21d409c2e24947beb99351da46a5aef9
1029 08:52:30.607040 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.607116 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2617559}
W20251029 08:52:30.608212 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 21d409c2e24947beb99351da46a5aef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d409c2e24947beb99351da46a5aef9 already in progress: creating tablet
I20251029 08:52:30.608645 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1139) took 2404 ms. Trace:
I20251029 08:52:30.609025 1179 rpcz_store.cc:276] 1029 08:52:28.204370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.204423 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:30.608545 (+2404122us) ts_tablet_manager.cc:1151] Deleting tablet 56f9308235f145ee838a55434f772bfa
1029 08:52:30.608555 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.608629 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2403863}
W20251029 08:52:30.609640 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 56f9308235f145ee838a55434f772bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f9308235f145ee838a55434f772bfa already in progress: creating tablet
I20251029 08:52:30.610071 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1140) took 2367 ms. Trace:
I20251029 08:52:30.610419 1179 rpcz_store.cc:276] 1029 08:52:28.242913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.243072 (+ 159us) service_pool.cc:225] Handling call
1029 08:52:30.609975 (+2366903us) ts_tablet_manager.cc:1151] Deleting tablet bdfda0866a614165824d1a3f7ad89af7
1029 08:52:30.609984 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.610056 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366655}
I20251029 08:52:30.611035 2302 ts_manager.cc:194] Registered new tserver with Master: 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:30.614243 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet bdfda0866a614165824d1a3f7ad89af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdfda0866a614165824d1a3f7ad89af7 already in progress: creating tablet
I20251029 08:52:30.614733 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1141) took 2368 ms. Trace:
I20251029 08:52:30.615190 1179 rpcz_store.cc:276] 1029 08:52:28.246407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.246699 (+ 292us) service_pool.cc:225] Handling call
1029 08:52:30.614590 (+2367891us) ts_tablet_manager.cc:1151] Deleting tablet 420c3357ebff4dd08df4ed2477b40ddf
1029 08:52:30.614601 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.614712 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367704}
W20251029 08:52:30.615928 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 420c3357ebff4dd08df4ed2477b40ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420c3357ebff4dd08df4ed2477b40ddf already in progress: creating tablet
I20251029 08:52:30.616479 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1142) took 2332 ms. Trace:
I20251029 08:52:30.616905 1179 rpcz_store.cc:276] 1029 08:52:28.283576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.283660 (+ 84us) service_pool.cc:225] Handling call
1029 08:52:30.616347 (+2332687us) ts_tablet_manager.cc:1151] Deleting tablet c5d9527d294c432db2ccf60af84731b0
1029 08:52:30.616359 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.616461 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332484}
W20251029 08:52:30.617563 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c5d9527d294c432db2ccf60af84731b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d9527d294c432db2ccf60af84731b0 already in progress: creating tablet
I20251029 08:52:30.618045 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1143) took 2334 ms. Trace:
I20251029 08:52:30.618432 1179 rpcz_store.cc:276] 1029 08:52:28.283726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.283896 (+ 170us) service_pool.cc:225] Handling call
1029 08:52:30.617926 (+2334030us) ts_tablet_manager.cc:1151] Deleting tablet 5e35d40b730a4a7883c324a6cc93b161
1029 08:52:30.617935 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.618029 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333870}
W20251029 08:52:30.619063 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5e35d40b730a4a7883c324a6cc93b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e35d40b730a4a7883c324a6cc93b161 already in progress: creating tablet
I20251029 08:52:30.619596 1179 tablet_replica.cc:333] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.603462 2707 leader_election.cc:290] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.620008 1179 raft_consensus.cc:2243] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.620515 1179 raft_consensus.cc:2272] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.621271 1179 ts_tablet_manager.cc:1916] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.624286 745 tablet_service.cc:1505] Processing CreateTablet for tablet 69646d1c8eb843ec82f25cbd7aa65849 (DEFAULT_TABLE table=default.Table_b2c950d7747b4bfebfb385e8fb2f8d59 [id=9c9db0fbaf4940e0a56469c15a806c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.624745 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69646d1c8eb843ec82f25cbd7aa65849. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:30.628381 998 leader_election.cc:343] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: eeabf8f23751447aa9af02363c40361d
I20251029 08:52:30.628379 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:30.628528 998 leader_election.cc:304] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.628573 774 raft_consensus.cc:3060] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:30.628978 998 leader_election.cc:343] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: f5b52314139945dd9c2e6ec899bf06dc
I20251029 08:52:30.629074 998 leader_election.cc:304] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.629444 2707 raft_consensus.cc:2749] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.629685 2707 raft_consensus.cc:2749] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.631480 774 raft_consensus.cc:2468] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:30.637748 998 leader_election.cc:304] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:30.638164 2797 raft_consensus.cc:2804] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:30.638339 2797 raft_consensus.cc:697] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:30.638648 2797 consensus_queue.cc:237] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.645677 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 69646d1c8eb843ec82f25cbd7aa65849 (DEFAULT_TABLE table=default.Table_b2c950d7747b4bfebfb385e8fb2f8d59 [id=9c9db0fbaf4940e0a56469c15a806c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.646416 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69646d1c8eb843ec82f25cbd7aa65849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.649816 1179 ts_tablet_manager.cc:1929] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.650388 1179 log.cc:1199] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0ab56045bd814d59831a98618c759af4
I20251029 08:52:30.651049 1179 ts_tablet_manager.cc:1950] T 0ab56045bd814d59831a98618c759af4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.652446 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1146) took 2299 ms. Trace:
I20251029 08:52:30.653471 1179 rpcz_store.cc:276] 1029 08:52:28.353187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.353242 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:30.619546 (+2266304us) ts_tablet_manager.cc:1151] Deleting tablet 0ab56045bd814d59831a98618c759af4
1029 08:52:30.619555 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.633111 (+ 13556us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.649693 (+ 16582us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.652425 (+ 2732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266080}
I20251029 08:52:30.654258 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0ab56045bd814d59831a98618c759af4 (table default.Table_29abde71edfc496a8c61702f28f25c63 [id=771dd3dd3a224a8599ab86bc7801dfa3]) successfully deleted
I20251029 08:52:30.654742 1179 tablet_replica.cc:333] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.655189 1179 raft_consensus.cc:2243] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.655652 1179 raft_consensus.cc:2272] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.656349 1179 ts_tablet_manager.cc:1916] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.656679 1183 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.656813 1183 ts_tablet_manager.cc:1403] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.164s user 0.014s sys 0.030s
I20251029 08:52:30.657606 1183 raft_consensus.cc:359] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.657773 1183 raft_consensus.cc:385] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.657821 1183 raft_consensus.cc:740] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.658002 1183 consensus_queue.cc:260] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.658473 1183 ts_tablet_manager.cc:1434] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:30.658735 1183 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.660178 1176 ts_tablet_manager.cc:1929] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.660319 1176 log.cc:1199] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/85df43bf781c421688f163331de0eb2b
I20251029 08:52:30.661715 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d87191d05d4145a796a8823f723bc499
W20251029 08:52:30.662878 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 980a0c3a08834c2484d7964d88f17bb0 consensus state has no leader
I20251029 08:52:30.663481 1183 tablet_bootstrap.cc:654] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.664217 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7492195a41664ef69554d8a6cb148e21
I20251029 08:52:30.664849 592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_620a010c6d044d05af5e6a82066c21c3" } modify_external_catalogs: true
I20251029 08:52:30.665014 592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_620a010c6d044d05af5e6a82066c21c3" } modify_external_catalogs: true
I20251029 08:52:30.665058 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 7492195a41664ef69554d8a6cb148e21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.665750 589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_54ce98414ac0405db8a6149c47af342c" } modify_external_catalogs: true
I20251029 08:52:30.665891 589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_54ce98414ac0405db8a6149c47af342c" } modify_external_catalogs: true
W20251029 08:52:30.666409 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a293b79308646358704e9c11836c81b consensus state has no leader
I20251029 08:52:30.666574 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet d87191d05d4145a796a8823f723bc499 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.667400 1047 tablet_service.cc:1505] Processing CreateTablet for tablet de61a37298ca465b9262f46c3b78cee5 (DEFAULT_TABLE table=default.Table_e522547c00dd4f70a3f03086eb6f372f [id=02238ce3f0cf4ac08c8fdef5595413bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.667847 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de61a37298ca465b9262f46c3b78cee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.668349 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet eeabf8f23751447aa9af02363c40361d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:30.668821 594 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d0cc7fe1c474a7d8bcc49be467f297c consensus state has no leader
I20251029 08:52:30.669199 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 9e44e56554574d5abb67d08d88218152 (DEFAULT_TABLE table=default.Table_4c77960427da4195879204c4bd7833a2 [id=3d0b5f416dda4e4f83173c1e9de7093c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.670130 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 4dec9fdf1ae94ae687f30f89096ce89b
I20251029 08:52:30.670737 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 59ad04f57c29407b9d2420520c095a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.671183 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.673123 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 07bf5d693be240f18bacccc81a6ea5d4
I20251029 08:52:30.673679 595 catalog_manager.cc:6719] ReplaceTablet: tablet 07bf5d693be240f18bacccc81a6ea5d4 deleted and replaced by tablet 32929763e9b74f3ebb689e6ab5828443
I20251029 08:52:30.674083 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 32929763e9b74f3ebb689e6ab5828443) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.674453 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e44e56554574d5abb67d08d88218152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.677228 2707 raft_consensus.cc:493] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:30.678787 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29cfe51f9a184a0280d26cd73660566b consensus state has no leader
I20251029 08:52:30.677541 2707 raft_consensus.cc:515] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.687353 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.613137 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:30.615554 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73a300a67f64cd6b6e9e9bb2e82db69. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:30.690335 996 leader_election.cc:343] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.695672 1179 ts_tablet_manager.cc:1929] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.695802 1179 log.cc:1199] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/6d6d720fb6b54b1e8bb7b3956c48b3f9
I20251029 08:52:30.696115 1179 ts_tablet_manager.cc:1950] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.699046 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "391977048e5f4f0b8f62e4d5e3d2946b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.699157 904 tablet_service.cc:1552] Processing DeleteTablet for tablet d87191d05d4145a796a8823f723bc499 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.650764 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1aaf233eba2461f9182e094a0115a4e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.700134 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fda3b7ac846dd8c05ced4eee5bd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.657488 1177 tablet_bootstrap.cc:492] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:30.667783 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 7492195a41664ef69554d8a6cb148e21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.668172 905 tablet_service.cc:1505] Processing CreateTablet for tablet de61a37298ca465b9262f46c3b78cee5 (DEFAULT_TABLE table=default.Table_e522547c00dd4f70a3f03086eb6f372f [id=02238ce3f0cf4ac08c8fdef5595413bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.702757 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de61a37298ca465b9262f46c3b78cee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.669435 896 tablet_service.cc:1505] Processing CreateTablet for tablet 9e44e56554574d5abb67d08d88218152 (DEFAULT_TABLE table=default.Table_4c77960427da4195879204c4bd7833a2 [id=3d0b5f416dda4e4f83173c1e9de7093c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.704283 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e44e56554574d5abb67d08d88218152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.707276 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69646d1c8eb843ec82f25cbd7aa65849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.672330 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.674999 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 32929763e9b74f3ebb689e6ab5828443) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.683285 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 09968d93203140fa9201928c87702ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.683429 888 tablet_service.cc:1552] Processing DeleteTablet for tablet bf29e030892d44e1ad4d1ca4f5aeab3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.709141 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1148) took 2336 ms. Trace:
I20251029 08:52:30.714424 1179 rpcz_store.cc:276] 1029 08:52:28.372214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.372264 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:30.654667 (+2282403us) ts_tablet_manager.cc:1151] Deleting tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9
1029 08:52:30.654679 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.665631 (+ 10952us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.695572 (+ 29941us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.709114 (+ 13542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2282202}
I20251029 08:52:30.714970 1179 tablet_replica.cc:333] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.715260 1179 raft_consensus.cc:2243] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.715489 1179 raft_consensus.cc:2272] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.716181 1179 ts_tablet_manager.cc:1916] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.689708 2731 raft_consensus.cc:493] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:30.716962 2731 raft_consensus.cc:515] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.717628 2731 leader_election.cc:290] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.698714 2707 leader_election.cc:290] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:30.699680 996 leader_election.cc:343] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.718396 2707 raft_consensus.cc:493] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 08:52:30.718573 996 leader_election.cc:343] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.718555 2707 raft_consensus.cc:515] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.719153 2707 leader_election.cc:290] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.719188 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 (table default.Table_0073b1df0b7d4647bff7d80fe246edf6 [id=b426ad17cef044b6aacc1c9da2714432]) successfully deleted
I20251029 08:52:30.720508 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59ad04f57c29407b9d2420520c095a28" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.721300 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bab242baf18b4ff8a3cb134178846f31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.721370 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_a6e6d2c56981480d877ea3030fcde485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.721885 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6e6d2c56981480d877ea3030fcde485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.722309 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b1be633f29d54c23b4a29d851377d3ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.722738 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1be633f29d54c23b4a29d851377d3ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:30.721892 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bab242baf18b4ff8a3cb134178846f31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.724488 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e229e75b9314413f8e7b3696ee9e6f55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.724888 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e229e75b9314413f8e7b3696ee9e6f55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.726404 1177 ts_tablet_manager.cc:1403] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.364s user 0.002s sys 0.002s
W20251029 08:52:30.719748 998 leader_election.cc:343] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 391977048e5f4f0b8f62e4d5e3d2946b
W20251029 08:52:30.721050 996 leader_election.cc:343] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.728739 998 leader_election.cc:304] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.728871 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_de20d4a8539d454fab53870dc689bc48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.729321 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de20d4a8539d454fab53870dc689bc48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.729403 2359 raft_consensus.cc:2749] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:30.729923 998 leader_election.cc:343] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 59ad04f57c29407b9d2420520c095a28
I20251029 08:52:30.730032 998 leader_election.cc:304] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.730432 2707 raft_consensus.cc:2749] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.727248 1177 raft_consensus.cc:359] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.732636 1177 raft_consensus.cc:385] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.724350 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b2024dec46a943a8a1e4866ada0a388e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:30.733189 1177 raft_consensus.cc:740] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
W20251029 08:52:30.733639 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2024dec46a943a8a1e4866ada0a388e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.734092 1177 consensus_queue.cc:260] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.735488 1177 ts_tablet_manager.cc:1434] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251029 08:52:30.738606 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ac317ca8b45f460f88ecb16a1bf0d802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.739238 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac317ca8b45f460f88ecb16a1bf0d802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.744415 1179 ts_tablet_manager.cc:1929] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.744494 905 tablet_service.cc:1552] Processing DeleteTablet for tablet 657cdaad97e24611a997e26a6293ea5e with delete_type TABLET_DATA_DELETED (replaced by tablet a506e35afe6a4c87901f27c434f3b348) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.744642 1179 log.cc:1199] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:30.745074 1179 ts_tablet_manager.cc:1950] T 9429f6abb50c4f66b2e1b565f303962d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.745996 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1147) took 2374 ms. Trace:
I20251029 08:52:30.746141 1179 rpcz_store.cc:276] 1029 08:52:28.371750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.373480 (+ 1730us) service_pool.cc:225] Handling call
1029 08:52:30.714915 (+2341435us) ts_tablet_manager.cc:1151] Deleting tablet 9429f6abb50c4f66b2e1b565f303962d
1029 08:52:30.714926 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.726102 (+ 11176us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.729069 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.745977 (+ 16908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2341228}
W20251029 08:52:30.672343 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 77696.2us
I20251029 08:52:30.746659 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9429f6abb50c4f66b2e1b565f303962d (table default.Table_a5e58cbcddc94f118f224404b14c6790 [id=04da6a1d6cc749dab385c6b7af982bec]) successfully deleted
I20251029 08:52:30.747087 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1157) took 2242 ms. Trace:
I20251029 08:52:30.747217 1179 rpcz_store.cc:276] 1029 08:52:28.504455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.504513 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:30.746949 (+2242436us) ts_tablet_manager.cc:1151] Deleting tablet e14d1d3421a542ec960d21453bb713ec
1029 08:52:30.746960 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.747071 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2242240}
W20251029 08:52:30.749819 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e14d1d3421a542ec960d21453bb713ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d1d3421a542ec960d21453bb713ec already in progress: creating tablet
I20251029 08:52:30.750288 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1171) took 1636 ms. Trace:
I20251029 08:52:30.750460 1179 rpcz_store.cc:276] 1029 08:52:29.113988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.114054 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:30.750154 (+1636100us) ts_tablet_manager.cc:1151] Deleting tablet d1d2df9e1e744ed1a6cf549ab57ed266
1029 08:52:30.750165 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.750269 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621488}
W20251029 08:52:30.750762 861 leader_election.cc:343] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 6689cf0804f946c18ebc555a191549d4
I20251029 08:52:30.751256 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 73db194649ac4d8caea12c56c67f78d0 (DEFAULT_TABLE table=default.Table_6fe654db95e54d37a885e5e31bc35b37 [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.751662 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db194649ac4d8caea12c56c67f78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.751772 896 tablet_service.cc:1505] Processing CreateTablet for tablet 36e1c037c1354e2c82be57f6310a421d (DEFAULT_TABLE table=default.Table_02ce044326e84d9f807967a0a3e56020 [id=1f6d1e4b812a411fbc066f98442d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.752003 861 leader_election.cc:304] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:30.753050 2731 raft_consensus.cc:2749] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.758087 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_f0938163b61540419aa4d8be600dcc1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.758651 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0938163b61540419aa4d8be600dcc1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:30.758944 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d1d2df9e1e744ed1a6cf549ab57ed266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d2df9e1e744ed1a6cf549ab57ed266 already in progress: creating tablet
I20251029 08:52:30.759352 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1172) took 1555 ms. Trace:
I20251029 08:52:30.759819 1179 rpcz_store.cc:276] 1029 08:52:29.203961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.204044 (+ 83us) service_pool.cc:225] Handling call
1029 08:52:30.759233 (+1555189us) ts_tablet_manager.cc:1151] Deleting tablet c9b611ae08a247d3ba02eb177c6b1b12
1029 08:52:30.759243 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.759333 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554938}
W20251029 08:52:30.760313 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c9b611ae08a247d3ba02eb177c6b1b12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b611ae08a247d3ba02eb177c6b1b12 already in progress: creating tablet
I20251029 08:52:30.761929 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1173) took 1485 ms. Trace:
I20251029 08:52:30.762053 1179 rpcz_store.cc:276] 1029 08:52:29.276853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.276901 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:30.761818 (+1484917us) ts_tablet_manager.cc:1151] Deleting tablet b61829fcd59d4ad4ab427270b7089d57
1029 08:52:30.761827 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.761911 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437535}
I20251029 08:52:30.762305 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1035) took 5090 ms. Trace:
W20251029 08:52:30.762507 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b61829fcd59d4ad4ab427270b7089d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61829fcd59d4ad4ab427270b7089d57 already in progress: creating tablet
I20251029 08:52:30.762861 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1176) took 1056 ms. Trace:
I20251029 08:52:30.762980 1179 rpcz_store.cc:276] 1029 08:52:29.706589 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.706629 (+ 40us) service_pool.cc:225] Handling call
1029 08:52:30.762755 (+1056126us) ts_tablet_manager.cc:1151] Deleting tablet 297945c50119432a9f9d3e356d573879
1029 08:52:30.762765 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.762844 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055939}
W20251029 08:52:30.763453 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 297945c50119432a9f9d3e356d573879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 297945c50119432a9f9d3e356d573879 already in progress: creating tablet
I20251029 08:52:30.763975 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1177) took 1057 ms. Trace:
I20251029 08:52:30.764484 1974 rpcz_store.cc:276] 1029 08:52:25.672049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.672088 (+ 39us) service_pool.cc:225] Handling call
1029 08:52:30.365833 (+4693745us) ts_tablet_manager.cc:1151] Deleting tablet 652903cbbdb042d780e7d71fc2c0ab4d
1029 08:52:30.365845 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.550135 (+184290us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.597899 (+ 47764us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.762283 (+164384us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51353,"tablet-delete.queue_time_us":4693628}
I20251029 08:52:30.665529 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 7492195a41664ef69554d8a6cb148e21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.666927 2786 raft_consensus.cc:493] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.765247 2786 raft_consensus.cc:515] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.765879 2786 leader_election.cc:290] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:30.670156 750 tablet_service.cc:1505] Processing CreateTablet for tablet 9e44e56554574d5abb67d08d88218152 (DEFAULT_TABLE table=default.Table_4c77960427da4195879204c4bd7833a2 [id=3d0b5f416dda4e4f83173c1e9de7093c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.766386 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e44e56554574d5abb67d08d88218152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.766861 1179 rpcz_store.cc:276] 1029 08:52:29.706694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.706849 (+ 155us) service_pool.cc:225] Handling call
1029 08:52:30.763871 (+1057022us) ts_tablet_manager.cc:1151] Deleting tablet 5d292c29d2b34086a5976da251f9ed5e
1029 08:52:30.763879 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.763959 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056858}
W20251029 08:52:30.767262 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5d292c29d2b34086a5976da251f9ed5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d292c29d2b34086a5976da251f9ed5e already in progress: creating tablet
I20251029 08:52:30.767611 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1178) took 1060 ms. Trace:
I20251029 08:52:30.767683 1179 rpcz_store.cc:276] 1029 08:52:29.706781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.706839 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:30.767499 (+1060660us) ts_tablet_manager.cc:1151] Deleting tablet adba5919a81348bb94569ca52fa4bded
1029 08:52:30.767509 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.767594 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1060437}
W20251029 08:52:30.768101 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet adba5919a81348bb94569ca52fa4bded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adba5919a81348bb94569ca52fa4bded already in progress: creating tablet
I20251029 08:52:30.770291 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 8734814cf3a24a7bb460f8caf862a4f7 (DEFAULT_TABLE table=default.Table_479cc5f9383943c49d34e88baf0b5b28 [id=12705f5eda8f45a18fbb06b8169e8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.770684 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8734814cf3a24a7bb460f8caf862a4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.671523 758 tablet_service.cc:1505] Processing CreateTablet for tablet 73db194649ac4d8caea12c56c67f78d0 (DEFAULT_TABLE table=default.Table_6fe654db95e54d37a885e5e31bc35b37 [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.771952 752 tablet_service.cc:1505] Processing CreateTablet for tablet 8734814cf3a24a7bb460f8caf862a4f7 (DEFAULT_TABLE table=default.Table_479cc5f9383943c49d34e88baf0b5b28 [id=12705f5eda8f45a18fbb06b8169e8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.772148 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db194649ac4d8caea12c56c67f78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.772294 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8734814cf3a24a7bb460f8caf862a4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.776518 1050 tablet_service.cc:1505] Processing CreateTablet for tablet e2c7388debdf4a0da6130a385a495245 (DEFAULT_TABLE table=default.Table_c08dc513f0834dd2b095f48146990078 [id=f5f2f640dc494c6084e95962a131099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.776886 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c7388debdf4a0da6130a385a495245. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.778419 1032 tablet_service.cc:1505] Processing CreateTablet for tablet ec30395554db46c2bae8572f8e810d1a (DEFAULT_TABLE table=default.Table_533c60a10a78409e9fd7e95fd47ecdd8 [id=457dfeeb417b4e2fb49240829235c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.672070 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dec9fdf1ae94ae687f30f89096ce89b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.778782 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec30395554db46c2bae8572f8e810d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.779049 764 tablet_service.cc:1505] Processing CreateTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 (DEFAULT_TABLE table=default.Table_ea2dc34a3418478cbf34a81ec9dd33e1 [id=cd6b6e94e10348969d9f4b766c166e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.779425 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79ec90d93ee48abaaed3bc3cd41bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.780457 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 36e1c037c1354e2c82be57f6310a421d (DEFAULT_TABLE table=default.Table_02ce044326e84d9f807967a0a3e56020 [id=1f6d1e4b812a411fbc066f98442d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.780764 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1c037c1354e2c82be57f6310a421d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.782301 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5bb0fd4c62d47a2b10719ebd55851c0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:30.714015 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 07bf5d693be240f18bacccc81a6ea5d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 32929763e9b74f3ebb689e6ab5828443) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.783227 753 tablet_service.cc:1505] Processing CreateTablet for tablet ec30395554db46c2bae8572f8e810d1a (DEFAULT_TABLE table=default.Table_533c60a10a78409e9fd7e95fd47ecdd8 [id=457dfeeb417b4e2fb49240829235c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.783339 752 tablet_service.cc:1505] Processing CreateTablet for tablet e2c7388debdf4a0da6130a385a495245 (DEFAULT_TABLE table=default.Table_c08dc513f0834dd2b095f48146990078 [id=f5f2f640dc494c6084e95962a131099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.783470 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0e7d5c552d4685b76666856a02a11c (DEFAULT_TABLE table=default.Table_153c84094e934e56abac30a2f8d6d02b [id=3287b7a43e1f4cb28c29bf81791a95d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.783568 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec30395554db46c2bae8572f8e810d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.783756 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c7388debdf4a0da6130a385a495245. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.783763 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e7d5c552d4685b76666856a02a11c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:30.790956 720 leader_election.cc:343] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.801074 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 173132d8c3af4b5a874f51bd8ba61b68 (DEFAULT_TABLE table=default.Table_4a8284ff6d314bbc976db77d1cb42d29 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.801514 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173132d8c3af4b5a874f51bd8ba61b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.804548 1047 tablet_service.cc:1505] Processing CreateTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 (DEFAULT_TABLE table=default.Table_ea2dc34a3418478cbf34a81ec9dd33e1 [id=cd6b6e94e10348969d9f4b766c166e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.804998 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79ec90d93ee48abaaed3bc3cd41bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.808296 592 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e322478c9ba64c2a95a2c299123926dd
I20251029 08:52:30.808687 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_47d202b2ab454856a15f3b5705fc9188" } new_table_name: "default.Table_19963a3f1dbd45ceab6c86ad0efe7377" modify_external_catalogs: true
I20251029 08:52:30.809058 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6a613283962a4289a60037f57d6c1e1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.809571 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a613283962a4289a60037f57d6c1e1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.813390 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 12e3a6ed1d1848af9b266a254571e872 (DEFAULT_TABLE table=default.Table_26a3af4de195431a994db99495fdbeb2 [id=871ffedb5be7460ba4f0c334e05828d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.813488 752 tablet_service.cc:1505] Processing CreateTablet for tablet 12e3a6ed1d1848af9b266a254571e872 (DEFAULT_TABLE table=default.Table_26a3af4de195431a994db99495fdbeb2 [id=871ffedb5be7460ba4f0c334e05828d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.813778 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e3a6ed1d1848af9b266a254571e872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.813870 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e3a6ed1d1848af9b266a254571e872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.818348 589 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 6dc6c8f9ee9840c08c9436ab516a62d7
I20251029 08:52:30.819056 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.820273 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1179) took 1113 ms. Trace:
I20251029 08:52:30.820423 1179 rpcz_store.cc:276] 1029 08:52:29.706891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.706929 (+ 38us) service_pool.cc:225] Handling call
1029 08:52:30.820144 (+1113215us) ts_tablet_manager.cc:1151] Deleting tablet 76dcdd90e4a54cbfb0cb62192c465c40
1029 08:52:30.820155 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.820255 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113050}
I20251029 08:52:30.820871 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1175) took 1114 ms. Trace:
I20251029 08:52:30.821005 1179 rpcz_store.cc:276] 1029 08:52:29.706411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.706943 (+ 532us) service_pool.cc:225] Handling call
1029 08:52:30.820770 (+1113827us) ts_tablet_manager.cc:1151] Deleting tablet b79ec90d93ee48abaaed3bc3cd41bf62
1029 08:52:30.820778 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.820855 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113566}
I20251029 08:52:30.821489 1179 tablet_replica.cc:333] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.821723 1179 raft_consensus.cc:2243] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:30.821743 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 76dcdd90e4a54cbfb0cb62192c465c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76dcdd90e4a54cbfb0cb62192c465c40 already in progress: creating tablet
W20251029 08:52:30.821983 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet b79ec90d93ee48abaaed3bc3cd41bf62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b79ec90d93ee48abaaed3bc3cd41bf62 already in progress: creating tablet
I20251029 08:52:30.822170 1179 pending_rounds.cc:70] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:30.822258 1179 pending_rounds.cc:77] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037681928560640 op_type: NO_OP noop_request { }
I20251029 08:52:30.822417 1179 raft_consensus.cc:2889] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:30.822768 1179 raft_consensus.cc:2272] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.823884 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.824273 764 tablet_service.cc:1552] Processing DeleteTablet for tablet e322478c9ba64c2a95a2c299123926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.824481 1179 ts_tablet_manager.cc:1916] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.727856 745 tablet_service.cc:1505] Processing CreateTablet for tablet de61a37298ca465b9262f46c3b78cee5 (DEFAULT_TABLE table=default.Table_e522547c00dd4f70a3f03086eb6f372f [id=02238ce3f0cf4ac08c8fdef5595413bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.825819 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de61a37298ca465b9262f46c3b78cee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.759788 757 tablet_service.cc:1505] Processing CreateTablet for tablet 36e1c037c1354e2c82be57f6310a421d (DEFAULT_TABLE table=default.Table_02ce044326e84d9f807967a0a3e56020 [id=1f6d1e4b812a411fbc066f98442d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.783253 746 tablet_service.cc:1505] Processing CreateTablet for tablet 173132d8c3af4b5a874f51bd8ba61b68 (DEFAULT_TABLE table=default.Table_4a8284ff6d314bbc976db77d1cb42d29 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.828411 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1c037c1354e2c82be57f6310a421d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.681600 1178 tablet_bootstrap.cc:492] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:30.783253 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0e7d5c552d4685b76666856a02a11c (DEFAULT_TABLE table=default.Table_153c84094e934e56abac30a2f8d6d02b [id=3287b7a43e1f4cb28c29bf81791a95d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.828629 1178 ts_tablet_manager.cc:1403] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.315s user 0.001s sys 0.003s
I20251029 08:52:30.828799 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e7d5c552d4685b76666856a02a11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.829317 1178 raft_consensus.cc:359] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.829483 1178 raft_consensus.cc:385] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.829538 1178 raft_consensus.cc:740] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.829722 1178 consensus_queue.cc:260] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:30.830209 1178 ts_tablet_manager.cc:1434] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:30.830468 1178 tablet_bootstrap.cc:492] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:30.831184 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173132d8c3af4b5a874f51bd8ba61b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.835175 1178 tablet_bootstrap.cc:654] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.731348 749 tablet_service.cc:1552] Processing DeleteTablet for tablet d87191d05d4145a796a8823f723bc499 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.843127 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet e322478c9ba64c2a95a2c299123926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.660676 1176 ts_tablet_manager.cc:1950] T 85df43bf781c421688f163331de0eb2b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:30.799059 2554 raft_consensus.cc:493] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:30.851684 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 652903cbbdb042d780e7d71fc2c0ab4d (table default.Table_35877e0584a84248a87841ffcb0cbf94 [id=0e0c97d7ced7490b86425ea2e2e4530c]) successfully deleted
I20251029 08:52:30.851629 2554 raft_consensus.cc:515] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.815991 888 tablet_service.cc:1505] Processing CreateTablet for tablet 8734814cf3a24a7bb460f8caf862a4f7 (DEFAULT_TABLE table=default.Table_479cc5f9383943c49d34e88baf0b5b28 [id=12705f5eda8f45a18fbb06b8169e8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.816545 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0e7d5c552d4685b76666856a02a11c (DEFAULT_TABLE table=default.Table_153c84094e934e56abac30a2f8d6d02b [id=3287b7a43e1f4cb28c29bf81791a95d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.816947 895 tablet_service.cc:1505] Processing CreateTablet for tablet e2c7388debdf4a0da6130a385a495245 (DEFAULT_TABLE table=default.Table_c08dc513f0834dd2b095f48146990078 [id=f5f2f640dc494c6084e95962a131099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.817323 894 tablet_service.cc:1505] Processing CreateTablet for tablet ec30395554db46c2bae8572f8e810d1a (DEFAULT_TABLE table=default.Table_533c60a10a78409e9fd7e95fd47ecdd8 [id=457dfeeb417b4e2fb49240829235c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.853044 2554 leader_election.cc:290] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.817727 891 tablet_service.cc:1505] Processing CreateTablet for tablet 173132d8c3af4b5a874f51bd8ba61b68 (DEFAULT_TABLE table=default.Table_4a8284ff6d314bbc976db77d1cb42d29 [id=5a81245c63184c9b9f5246f17d6e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.818077 901 tablet_service.cc:1505] Processing CreateTablet for tablet b79ec90d93ee48abaaed3bc3cd41bf62 (DEFAULT_TABLE table=default.Table_ea2dc34a3418478cbf34a81ec9dd33e1 [id=cd6b6e94e10348969d9f4b766c166e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.829655 905 tablet_service.cc:1505] Processing CreateTablet for tablet 73db194649ac4d8caea12c56c67f78d0 (DEFAULT_TABLE table=default.Table_6fe654db95e54d37a885e5e31bc35b37 [id=c488f3afc154469cab70bf8007773b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.838635 897 tablet_service.cc:1505] Processing CreateTablet for tablet 12e3a6ed1d1848af9b266a254571e872 (DEFAULT_TABLE table=default.Table_26a3af4de195431a994db99495fdbeb2 [id=871ffedb5be7460ba4f0c334e05828d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.838814 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.839084 893 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef24005087a450ca047c90502fcfdfd (DEFAULT_TABLE table=default.Table_5c1854f1ed1c420bb4402bc4e8d53b42 [id=5d155a4ad99c4c718be01fc51fe97d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.839457 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5bb0fd4c62d47a2b10719ebd55851c0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:30.854072 718 leader_election.cc:343] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.854182 718 leader_election.cc:304] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.854600 1179 ts_tablet_manager.cc:1929] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.854640 2786 raft_consensus.cc:2749] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.854714 1179 log.cc:1199] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/fcaa3fa779824b38887ae001623ba67b
I20251029 08:52:30.839968 904 tablet_service.cc:1552] Processing DeleteTablet for tablet e322478c9ba64c2a95a2c299123926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:30.855046 1179 ts_tablet_manager.cc:1950] T fcaa3fa779824b38887ae001623ba67b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.855863 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1181) took 1141 ms. Trace:
I20251029 08:52:30.857563 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1172) took 1141 ms. Trace:
I20251029 08:52:30.857651 1176 rpcz_store.cc:276] 1029 08:52:29.715720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.715861 (+ 141us) service_pool.cc:225] Handling call
1029 08:52:30.550277 (+834416us) ts_tablet_manager.cc:1151] Deleting tablet 85df43bf781c421688f163331de0eb2b
1029 08:52:30.550283 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.634586 (+ 84303us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.659988 (+ 25402us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.857546 (+197558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":834275}
I20251029 08:52:30.858057 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 85df43bf781c421688f163331de0eb2b (table default.Table_7501d57fa16e41269af41cad333961d2 [id=2e09778b9b214426bdcd4070c9f0abca]) successfully deleted
I20251029 08:52:30.863227 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a506e35afe6a4c87901f27c434f3b348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.864820 1178 tablet_bootstrap.cc:492] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:30.864944 1178 ts_tablet_manager.cc:1403] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20251029 08:52:30.865624 1178 raft_consensus.cc:359] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.865828 1178 raft_consensus.cc:385] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.865892 1178 raft_consensus.cc:740] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.865880 1179 rpcz_store.cc:276] 1029 08:52:29.714434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.714724 (+ 290us) service_pool.cc:225] Handling call
1029 08:52:30.821449 (+1106725us) ts_tablet_manager.cc:1151] Deleting tablet fcaa3fa779824b38887ae001623ba67b
1029 08:52:30.821457 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.844895 (+ 23438us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.854482 (+ 9587us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.855839 (+ 1357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106582}
I20251029 08:52:30.866125 1178 consensus_queue.cc:260] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.866433 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1184) took 1151 ms. Trace:
I20251029 08:52:30.866499 2707 raft_consensus.cc:493] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.866250 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet fcaa3fa779824b38887ae001623ba67b (table default.Table_47a8da9054c74094bd0a2f92bf0dba27 [id=0d4abe2c0f3845378e6e26f82d973541]) successfully deleted
I20251029 08:52:30.866617 1178 ts_tablet_manager.cc:1434] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:30.866525 1179 rpcz_store.cc:276] 1029 08:52:29.714676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.714895 (+ 219us) service_pool.cc:225] Handling call
1029 08:52:30.866315 (+1151420us) ts_tablet_manager.cc:1151] Deleting tablet 3f25fe685db94f3ba2003bc7e417277c
1029 08:52:30.866325 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.866417 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151303}
I20251029 08:52:30.866664 2707 raft_consensus.cc:515] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:30.866837 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3f25fe685db94f3ba2003bc7e417277c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f25fe685db94f3ba2003bc7e417277c already in progress: creating tablet
I20251029 08:52:30.866888 1178 tablet_bootstrap.cc:492] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:30.866940 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1183) took 1152 ms. Trace:
I20251029 08:52:30.866998 1179 rpcz_store.cc:276] 1029 08:52:29.714593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.715031 (+ 438us) service_pool.cc:225] Handling call
1029 08:52:30.866859 (+1151828us) ts_tablet_manager.cc:1151] Deleting tablet 6689cf0804f946c18ebc555a191549d4
1029 08:52:30.866867 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.866926 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151719}
I20251029 08:52:30.867231 2707 leader_election.cc:290] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.867261 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1180) took 1152 ms. Trace:
I20251029 08:52:30.867323 1179 rpcz_store.cc:276] 1029 08:52:29.714330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.715052 (+ 722us) service_pool.cc:225] Handling call
1029 08:52:30.867173 (+1152121us) ts_tablet_manager.cc:1151] Deleting tablet 8017e8fe5f634ef0bf7f8be7c8aea033
1029 08:52:30.867181 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.867249 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152022}
I20251029 08:52:30.867558 1179 tablet_replica.cc:333] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.867733 1179 raft_consensus.cc:2243] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.867869 1179 raft_consensus.cc:2272] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.868228 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9907d044daa145168417371c2bd66544" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:30.868412 1179 ts_tablet_manager.cc:1916] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:30.868472 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6689cf0804f946c18ebc555a191549d4 because the tablet was not found. No further retry: Not found: Tablet not found: 6689cf0804f946c18ebc555a191549d4
W20251029 08:52:30.868685 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8017e8fe5f634ef0bf7f8be7c8aea033 already in progress: creating tablet
W20251029 08:52:30.868774 998 leader_election.cc:343] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9907d044daa145168417371c2bd66544
W20251029 08:52:30.868928 996 leader_election.cc:343] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:30.869032 996 leader_election.cc:304] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:30.869355 2707 raft_consensus.cc:2749] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:30.872810 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f7ad1f9e93c45f782618995a8dda857" } modify_external_catalogs: true
I20251029 08:52:30.872970 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_4f7ad1f9e93c45f782618995a8dda857" } modify_external_catalogs: true
I20251029 08:52:30.874329 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_19667d766eb94067b16badc787433f2c" } new_table_name: "default.Table_fa532317de53450b9330eedb3260b1e8" modify_external_catalogs: true
I20251029 08:52:30.875485 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c312fa246a2c4435b6a2c1fa67420ec0" } new_table_name: "default.Table_557b79e1b3a6437ea857d2553b6ffcad" modify_external_catalogs: true
I20251029 08:52:30.877241 1179 ts_tablet_manager.cc:1929] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.877344 1179 log.cc:1199] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e3c24c0d32f74435bd9d28f2415e2c47
I20251029 08:52:30.877632 1179 ts_tablet_manager.cc:1950] T e3c24c0d32f74435bd9d28f2415e2c47 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.878473 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1182) took 1163 ms. Trace:
I20251029 08:52:30.878568 1179 rpcz_store.cc:276] 1029 08:52:29.714511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.715165 (+ 654us) service_pool.cc:225] Handling call
1029 08:52:30.867503 (+1152338us) ts_tablet_manager.cc:1151] Deleting tablet e3c24c0d32f74435bd9d28f2415e2c47
1029 08:52:30.867523 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.873719 (+ 6196us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.877145 (+ 3426us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.878451 (+ 1306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152221}
I20251029 08:52:30.879005 1179 tablet_replica.cc:333] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.879200 1179 raft_consensus.cc:2243] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:30.879283 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1938936fb1a5483c9d118a57d88d283b" } modify_external_catalogs: true
I20251029 08:52:30.879413 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_1938936fb1a5483c9d118a57d88d283b" } modify_external_catalogs: true
I20251029 08:52:30.879463 1179 pending_rounds.cc:70] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:30.879506 1179 pending_rounds.cc:77] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037680516796416 op_type: NO_OP noop_request { }
I20251029 08:52:30.879598 1179 raft_consensus.cc:2889] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:30.879671 1179 raft_consensus.cc:2272] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.880199 1179 ts_tablet_manager.cc:1916] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.880434 1178 tablet_bootstrap.cc:654] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.881170 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_17251c1e0e6447b0bb5b5cf8c78e42ed" } new_table_name: "default.Table_d0e2661feb2541078519a3f854f86d11" modify_external_catalogs: true
I20251029 08:52:30.882586 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_98334d1800134673a0ee67105e125d80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:30.883064 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98334d1800134673a0ee67105e125d80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:30.885218 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e3c24c0d32f74435bd9d28f2415e2c47 (table default.Table_3222378209e141649a30f5c940ae2c23 [id=b189fe1c508d4520a69d483cb97d4886]) successfully deleted
I20251029 08:52:30.885792 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_22ad2d235d9c4ccdb70814d03790c887" } new_table_name: "default.Table_407b5f5353eb41ac97c608ced9005d66" modify_external_catalogs: true
I20251029 08:52:30.886272 1179 ts_tablet_manager.cc:1929] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:30.886370 1179 log.cc:1199] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e0fa089fce564af4b03aa323cec6cbf6
I20251029 08:52:30.886724 1179 ts_tablet_manager.cc:1950] T e0fa089fce564af4b03aa323cec6cbf6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.887449 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1187) took 1089 ms. Trace:
I20251029 08:52:30.887560 1179 rpcz_store.cc:276] 1029 08:52:29.797843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.797894 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:30.878957 (+1081063us) ts_tablet_manager.cc:1151] Deleting tablet e0fa089fce564af4b03aa323cec6cbf6
1029 08:52:30.878967 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.883203 (+ 4236us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.886185 (+ 2982us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.887429 (+ 1244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080891}
I20251029 08:52:30.887964 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e0fa089fce564af4b03aa323cec6cbf6 (table default.Table_8d4ea5c42591424eb1be9e9e3981cc89 [id=1c0e2e0996134fe1811cb2e300a861ac]) successfully deleted
I20251029 08:52:30.888283 1179 tablet_replica.cc:333] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:30.889671 1179 raft_consensus.cc:2243] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:30.889851 1179 raft_consensus.cc:2272] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:30.893038 1183 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:30.893198 1183 ts_tablet_manager.cc:1403] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.235s user 0.003s sys 0.007s
I20251029 08:52:30.894152 1183 raft_consensus.cc:359] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.894394 1183 raft_consensus.cc:385] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.894444 1183 raft_consensus.cc:740] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.894688 1183 consensus_queue.cc:260] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.894744 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_038f3279c2364bdeb2dc381a25b699b5" } modify_external_catalogs: true
I20251029 08:52:30.895262 1183 ts_tablet_manager.cc:1434] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 08:52:30.895308 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_038f3279c2364bdeb2dc381a25b699b5" } modify_external_catalogs: true
I20251029 08:52:30.895718 1183 tablet_bootstrap.cc:492] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:30.897248 1183 tablet_bootstrap.cc:654] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.894170 1179 ts_tablet_manager.cc:1916] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:30.903785 1178 tablet_bootstrap.cc:492] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:30.903950 1178 ts_tablet_manager.cc:1403] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20251029 08:52:30.904647 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1c037c1354e2c82be57f6310a421d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.904724 1178 raft_consensus.cc:359] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.904906 1178 raft_consensus.cc:385] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:30.904971 1178 raft_consensus.cc:740] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:30.905172 1178 consensus_queue.cc:260] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.905687 1178 ts_tablet_manager.cc:1434] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:30.906997 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8734814cf3a24a7bb460f8caf862a4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.922669 1177 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:30.925122 749 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef24005087a450ca047c90502fcfdfd (DEFAULT_TABLE table=default.Table_5c1854f1ed1c420bb4402bc4e8d53b42 [id=5d155a4ad99c4c718be01fc51fe97d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.928311 1177 tablet_bootstrap.cc:654] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:30.929296 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0e7d5c552d4685b76666856a02a11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.932267 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c7388debdf4a0da6130a385a495245. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.936244 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec30395554db46c2bae8572f8e810d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.937342 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef24005087a450ca047c90502fcfdfd (DEFAULT_TABLE table=default.Table_5c1854f1ed1c420bb4402bc4e8d53b42 [id=5d155a4ad99c4c718be01fc51fe97d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.937943 746 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0cc7fe1c474a7d8bcc49be467f297c (DEFAULT_TABLE table=default.Table_538d99e1eab6407ea3f0899f8db94c96 [id=d742120c68004b65af7443c06a4f7d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.938570 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0cc7fe1c474a7d8bcc49be467f297c (DEFAULT_TABLE table=default.Table_538d99e1eab6407ea3f0899f8db94c96 [id=d742120c68004b65af7443c06a4f7d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.938951 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0cc7fe1c474a7d8bcc49be467f297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.939474 747 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3379481937491c8ba01dbccaf3cccd (DEFAULT_TABLE table=default.Table_7a2164d9430240f8a00e62963f6edb46 [id=9dce76500cf9455483b052eb3292aea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.939711 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef24005087a450ca047c90502fcfdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.940243 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173132d8c3af4b5a874f51bd8ba61b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.942059 1047 tablet_service.cc:1505] Processing CreateTablet for tablet d7c1465dced74a77943931d31a7ca667 (DEFAULT_TABLE table=default.Table_02232cd3f46941d3865a4159bf92dc0a [id=4d3698e5cdc646d28eb0776a8450f2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.942389 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c1465dced74a77943931d31a7ca667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.944296 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3379481937491c8ba01dbccaf3cccd (DEFAULT_TABLE table=default.Table_7a2164d9430240f8a00e62963f6edb46 [id=9dce76500cf9455483b052eb3292aea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.944653 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3379481937491c8ba01dbccaf3cccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.945890 904 tablet_service.cc:1505] Processing CreateTablet for tablet d7c1465dced74a77943931d31a7ca667 (DEFAULT_TABLE table=default.Table_02232cd3f46941d3865a4159bf92dc0a [id=4d3698e5cdc646d28eb0776a8450f2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.946234 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c1465dced74a77943931d31a7ca667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.946287 2845 raft_consensus.cc:493] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:30.946414 2845 raft_consensus.cc:515] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:30.947011 2845 leader_election.cc:290] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:30.953751 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79ec90d93ee48abaaed3bc3cd41bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.955493 2833 catalog_manager.cc:1269] Loaded cluster ID: ce6f29e3310a47db9578bdd5331fd09c
I20251029 08:52:30.955622 2833 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: loading cluster ID for follower catalog manager: success
I20251029 08:52:30.956226 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e3a6ed1d1848af9b266a254571e872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.958343 1179 ts_tablet_manager.cc:1929] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:30.958442 1179 log.cc:1199] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/391977048e5f4f0b8f62e4d5e3d2946b
I20251029 08:52:30.958719 1179 ts_tablet_manager.cc:1950] T 391977048e5f4f0b8f62e4d5e3d2946b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:30.960283 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef24005087a450ca047c90502fcfdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.965009 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db194649ac4d8caea12c56c67f78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:30.975023 886 tablet_service.cc:1505] Processing CreateTablet for tablet 0c3379481937491c8ba01dbccaf3cccd (DEFAULT_TABLE table=default.Table_7a2164d9430240f8a00e62963f6edb46 [id=9dce76500cf9455483b052eb3292aea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:30.975467 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3379481937491c8ba01dbccaf3cccd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:30.981177 588 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00829c7fa0fe4e379a1a3044550e5941 consensus state has no leader
W20251029 08:52:30.981947 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf71cd77855841b5a914a3056a130112 consensus state has no leader
I20251029 08:52:30.983161 597 master_service.cc:954] ReplaceTablet: received request to replace tablet e1e2be77182e4259b76d20cae4e9e478 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:30.983562 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 0b0f1cdfb1804d1c8d6220907795fc80
I20251029 08:52:30.984458 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:30.985011 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:30.987795 2833 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: acquiring CA information for follower catalog manager: success
I20251029 08:52:31.001230 891 tablet_service.cc:1505] Processing CreateTablet for tablet 1a293b79308646358704e9c11836c81b (DEFAULT_TABLE table=default.Table_6090d35adbbd4757b87805278db68aa5 [id=f7b1d697e7924a7fa83f41f0a19f29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.001791 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a293b79308646358704e9c11836c81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.008680 753 tablet_service.cc:1505] Processing CreateTablet for tablet d7c1465dced74a77943931d31a7ca667 (DEFAULT_TABLE table=default.Table_02232cd3f46941d3865a4159bf92dc0a [id=4d3698e5cdc646d28eb0776a8450f2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.009559 764 tablet_service.cc:1505] Processing CreateTablet for tablet 1a293b79308646358704e9c11836c81b (DEFAULT_TABLE table=default.Table_6090d35adbbd4757b87805278db68aa5 [id=f7b1d697e7924a7fa83f41f0a19f29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.012157 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b0f1cdfb1804d1c8d6220907795fc80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.015067 2786 raft_consensus.cc:493] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.015210 2786 raft_consensus.cc:515] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:31.015851 2786 leader_election.cc:290] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:31.017186 893 tablet_service.cc:1505] Processing CreateTablet for tablet d1c3c731e79b4c3d9f2582c57a3b4899 (DEFAULT_TABLE table=default.Table_4db3fb50b14d461aad1b30f1f49045d0 [id=5c8da3783200413bbb48e4ba8010cf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.017234 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 1a293b79308646358704e9c11836c81b (DEFAULT_TABLE table=default.Table_6090d35adbbd4757b87805278db68aa5 [id=f7b1d697e7924a7fa83f41f0a19f29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.017591 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c3c731e79b4c3d9f2582c57a3b4899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.017666 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a293b79308646358704e9c11836c81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.018487 1047 tablet_service.cc:1505] Processing CreateTablet for tablet ae37fa6492d147bfbd564b734d38e1dd (DEFAULT_TABLE table=default.Table_6c82dc7545d54dc7995dc1a15f03d6b0 [id=6f26f9a84df8434ba0d5133884822d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.018818 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae37fa6492d147bfbd564b734d38e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.021031 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:31.021546 720 leader_election.cc:343] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.021801 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0cc7fe1c474a7d8bcc49be467f297c (DEFAULT_TABLE table=default.Table_538d99e1eab6407ea3f0899f8db94c96 [id=d742120c68004b65af7443c06a4f7d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.022131 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0cc7fe1c474a7d8bcc49be467f297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.022811 905 tablet_service.cc:1505] Processing CreateTablet for tablet ae37fa6492d147bfbd564b734d38e1dd (DEFAULT_TABLE table=default.Table_6c82dc7545d54dc7995dc1a15f03d6b0 [id=6f26f9a84df8434ba0d5133884822d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.023036 1974 tablet_replica.cc:333] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:31.023203 1974 raft_consensus.cc:2243] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.023350 1974 raft_consensus.cc:2272] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.023880 1974 ts_tablet_manager.cc:1916] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.024122 1046 tablet_service.cc:1505] Processing CreateTablet for tablet d1c3c731e79b4c3d9f2582c57a3b4899 (DEFAULT_TABLE table=default.Table_4db3fb50b14d461aad1b30f1f49045d0 [id=5c8da3783200413bbb48e4ba8010cf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.024436 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c3c731e79b4c3d9f2582c57a3b4899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.027842 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.028292 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef24005087a450ca047c90502fcfdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.028321 1178 tablet_bootstrap.cc:492] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.030311 1178 tablet_bootstrap.cc:654] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.030756 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0cc7fe1c474a7d8bcc49be467f297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.030947 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c3379481937491c8ba01dbccaf3cccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.031746 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae37fa6492d147bfbd564b734d38e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.035070 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1173) took 1319 ms. Trace:
W20251029 08:52:31.039587 718 leader_election.cc:343] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.039708 718 leader_election.cc:304] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.041716 750 tablet_service.cc:1505] Processing CreateTablet for tablet ae37fa6492d147bfbd564b734d38e1dd (DEFAULT_TABLE table=default.Table_6c82dc7545d54dc7995dc1a15f03d6b0 [id=6f26f9a84df8434ba0d5133884822d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.041915 2869 raft_consensus.cc:2749] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.042007 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c1465dced74a77943931d31a7ca667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.042752 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a293b79308646358704e9c11836c81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.035162 1176 rpcz_store.cc:276] 1029 08:52:29.715816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.716053 (+ 237us) service_pool.cc:225] Handling call
1029 08:52:31.034938 (+1318885us) ts_tablet_manager.cc:1151] Deleting tablet 3f25fe685db94f3ba2003bc7e417277c
1029 08:52:31.034953 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.035053 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318748}
I20251029 08:52:31.042328 758 tablet_service.cc:1505] Processing CreateTablet for tablet d1c3c731e79b4c3d9f2582c57a3b4899 (DEFAULT_TABLE table=default.Table_4db3fb50b14d461aad1b30f1f49045d0 [id=5c8da3783200413bbb48e4ba8010cf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.048454 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c3c731e79b4c3d9f2582c57a3b4899. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.048379 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3f25fe685db94f3ba2003bc7e417277c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f25fe685db94f3ba2003bc7e417277c already in progress: creating tablet
I20251029 08:52:31.051352 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae37fa6492d147bfbd564b734d38e1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.051934 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1189) took 1253 ms. Trace:
I20251029 08:52:31.052137 1179 rpcz_store.cc:276] 1029 08:52:29.798646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.798717 (+ 71us) service_pool.cc:225] Handling call
1029 08:52:30.888234 (+1089517us) ts_tablet_manager.cc:1151] Deleting tablet 391977048e5f4f0b8f62e4d5e3d2946b
1029 08:52:30.888245 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:30.955291 (+ 67046us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:30.958252 (+ 2961us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.051905 (+ 93653us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1191,"tablet-delete.queue_time_us":1089333}
I20251029 08:52:31.052634 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 391977048e5f4f0b8f62e4d5e3d2946b (table default.Table_cf4be2bf29e44a4bbb908e06223cc48d [id=48afa12a4fd24dc0918cd31e4da595b5]) successfully deleted
I20251029 08:52:31.052982 1179 tablet_replica.cc:333] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.053186 1179 raft_consensus.cc:2243] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.056133 1176 tablet_replica.cc:333] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:31.056325 1176 raft_consensus.cc:2243] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.056499 1176 raft_consensus.cc:2272] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.057047 1176 ts_tablet_manager.cc:1916] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.061347 891 tablet_service.cc:1505] Processing CreateTablet for tablet 7db62aa838ac4fa1a6cbc069f6c20de2 (DEFAULT_TABLE table=default.Table_c068416bd1584f84942bc94450654170 [id=c1264766ea1640b18453b08261607b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.065073 764 tablet_service.cc:1505] Processing CreateTablet for tablet 7db62aa838ac4fa1a6cbc069f6c20de2 (DEFAULT_TABLE table=default.Table_c068416bd1584f84942bc94450654170 [id=c1264766ea1640b18453b08261607b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.065552 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db62aa838ac4fa1a6cbc069f6c20de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.068176 2869 raft_consensus.cc:493] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.068354 2869 raft_consensus.cc:515] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:31.070835 1178 tablet_bootstrap.cc:492] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.070969 1178 ts_tablet_manager.cc:1403] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.005s
I20251029 08:52:31.071728 1178 raft_consensus.cc:359] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.071918 1178 raft_consensus.cc:385] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.071978 1178 raft_consensus.cc:740] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.069415 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a506e35afe6a4c87901f27c434f3b348" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.072199 1178 consensus_queue.cc:260] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.072535 1179 raft_consensus.cc:2272] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:31.072641 720 leader_election.cc:343] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.073140 1179 ts_tablet_manager.cc:1916] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.074681 1178 ts_tablet_manager.cc:1434] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:31.075441 1178 tablet_bootstrap.cc:492] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.081040 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 7db62aa838ac4fa1a6cbc069f6c20de2 (DEFAULT_TABLE table=default.Table_c068416bd1584f84942bc94450654170 [id=c1264766ea1640b18453b08261607b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.081457 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db62aa838ac4fa1a6cbc069f6c20de2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.082976 861 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
W20251029 08:52:31.084373 861 consensus_peers.cc:597] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ac1af8b78a4d94b0e3da14ebfffd92. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:31.084712 861 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726
W20251029 08:52:31.085305 861 consensus_peers.cc:597] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19fdc381a0d949b0b93cef399daa3021. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.088831 1178 tablet_bootstrap.cc:654] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.089381 2869 leader_election.cc:290] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:31.096349 750 tablet_service.cc:1505] Processing CreateTablet for tablet 29cfe51f9a184a0280d26cd73660566b (DEFAULT_TABLE table=default.Table_cfcd92d4157142e99ededb7b68d63193 [id=1f2d5a324f384b43bbfbc2c0b8eb9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.096781 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cfe51f9a184a0280d26cd73660566b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.097922 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 29cfe51f9a184a0280d26cd73660566b (DEFAULT_TABLE table=default.Table_cfcd92d4157142e99ededb7b68d63193 [id=1f2d5a324f384b43bbfbc2c0b8eb9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.098568 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cfe51f9a184a0280d26cd73660566b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.101927 998 consensus_peers.cc:597] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7046bb1fcdd34d308aecddfdf0752911. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.102849 1178 tablet_bootstrap.cc:492] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.102981 1178 ts_tablet_manager.cc:1403] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20251029 08:52:31.103732 1178 raft_consensus.cc:359] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.103900 1178 raft_consensus.cc:385] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.103958 1178 raft_consensus.cc:740] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.104158 1178 consensus_queue.cc:260] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.104702 1178 ts_tablet_manager.cc:1434] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251029 08:52:31.106048 996 consensus_peers.cc:597] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.106846 1178 tablet_bootstrap.cc:492] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.108502 1178 tablet_bootstrap.cc:654] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.111465 1051 tablet_service.cc:1505] Processing CreateTablet for tablet ffc33f4ac70e4aeda366a63fb178f6b2 (DEFAULT_TABLE table=default.Table_4680189b619a426596baa3e27ee29f57 [id=f5e60f8865dc43fdacd4e9c2f48cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.111936 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc33f4ac70e4aeda366a63fb178f6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.118896 1178 tablet_bootstrap.cc:492] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.119045 1178 ts_tablet_manager.cc:1403] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251029 08:52:31.121887 1178 raft_consensus.cc:359] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.122105 1178 raft_consensus.cc:385] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.122051 764 tablet_service.cc:1505] Processing CreateTablet for tablet ffc33f4ac70e4aeda366a63fb178f6b2 (DEFAULT_TABLE table=default.Table_4680189b619a426596baa3e27ee29f57 [id=f5e60f8865dc43fdacd4e9c2f48cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.122891 2833 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f718bd0f00004f20a89d028a2fa8e6b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 08:52:31.122157 1178 raft_consensus.cc:740] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.123286 1178 consensus_queue.cc:260] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.123785 1178 ts_tablet_manager.cc:1434] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 08:52:31.124110 1178 tablet_bootstrap.cc:492] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.125994 1178 tablet_bootstrap.cc:654] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.128068 2869 raft_consensus.cc:493] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.128257 2869 raft_consensus.cc:515] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.129287 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c90087c9e94665a9b3fac798304af7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.129614 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc33f4ac70e4aeda366a63fb178f6b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.129856 720 leader_election.cc:343] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.140341 2707 consensus_queue.cc:1048] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:31.149955 2869 leader_election.cc:290] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
W20251029 08:52:31.150719 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 65431.3us
W20251029 08:52:31.151021 861 leader_election.cc:343] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:31.151237 861 leader_election.cc:304] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
W20251029 08:52:31.151432 861 raft_consensus.cc:2686] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:31.151783 861 consensus_peers.cc:597] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ac1af8b78a4d94b0e3da14ebfffd92. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.091228 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a506e35afe6a4c87901f27c434f3b348" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.152420 718 leader_election.cc:343] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.152530 718 leader_election.cc:304] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.092355 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db62aa838ac4fa1a6cbc069f6c20de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.110612 886 tablet_service.cc:1505] Processing CreateTablet for tablet ffc33f4ac70e4aeda366a63fb178f6b2 (DEFAULT_TABLE table=default.Table_4680189b619a426596baa3e27ee29f57 [id=f5e60f8865dc43fdacd4e9c2f48cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.125041 897 tablet_service.cc:1505] Processing CreateTablet for tablet 4675f0aec6a14807afd3094780d60c72 (DEFAULT_TABLE table=default.Table_d2361c16d6164246a7b81385f8fcf1c6 [id=28b9344d4f194f42a186c230ea3153f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.134809 2845 raft_consensus.cc:493] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.161226 2845 raft_consensus.cc:515] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.161788 2845 leader_election.cc:290] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.162140 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4675f0aec6a14807afd3094780d60c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.162096 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 4675f0aec6a14807afd3094780d60c72 (DEFAULT_TABLE table=default.Table_d2361c16d6164246a7b81385f8fcf1c6 [id=28b9344d4f194f42a186c230ea3153f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.162431 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4675f0aec6a14807afd3094780d60c72. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.167332 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ad8a88123f8542eea86d8656bfef97df (table default.Table_07159bd6d35d4ba3b84c2c8c27ac779b [id=7a37f6e30a304e17a7a0ab5bd6b9564d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ad8a88123f8542eea86d8656bfef97df consensus state has no leader
I20251029 08:52:31.176373 2786 raft_consensus.cc:2749] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.178596 2888 tablet.cc:1722] T ec54e736a3e64dd7a839252c440f778c P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:31.183209 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc33f4ac70e4aeda366a63fb178f6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.135437 1177 tablet_bootstrap.cc:492] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:31.150367 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c90087c9e94665a9b3fac798304af7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.188766 1177 ts_tablet_manager.cc:1403] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.280s user 0.005s sys 0.000s
W20251029 08:52:31.188735 718 leader_election.cc:343] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.188876 718 leader_election.cc:304] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.189256 2786 raft_consensus.cc:2749] T 59c90087c9e94665a9b3fac798304af7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.189883 1177 raft_consensus.cc:359] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.090099 901 tablet_service.cc:1505] Processing CreateTablet for tablet 29cfe51f9a184a0280d26cd73660566b (DEFAULT_TABLE table=default.Table_cfcd92d4157142e99ededb7b68d63193 [id=1f2d5a324f384b43bbfbc2c0b8eb9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.190572 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cfe51f9a184a0280d26cd73660566b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.187084 1179 ts_tablet_manager.cc:1929] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.192363 1179 log.cc:1199] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/f5b52314139945dd9c2e6ec899bf06dc
I20251029 08:52:31.192785 1179 ts_tablet_manager.cc:1950] T f5b52314139945dd9c2e6ec899bf06dc P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.193843 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1190) took 1394 ms. Trace:
I20251029 08:52:31.194005 1179 rpcz_store.cc:276] 1029 08:52:29.799057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.799100 (+ 43us) service_pool.cc:225] Handling call
1029 08:52:31.052939 (+1253839us) ts_tablet_manager.cc:1151] Deleting tablet f5b52314139945dd9c2e6ec899bf06dc
1029 08:52:31.052948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.150559 (+ 97611us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.186964 (+ 36405us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.193821 (+ 6857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253717}
I20251029 08:52:31.194110 1178 tablet_bootstrap.cc:492] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.194308 1178 ts_tablet_manager.cc:1403] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.004s
I20251029 08:52:31.194722 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet f5b52314139945dd9c2e6ec899bf06dc (table default.Table_1a3b200aa88540f8b6dcbde2f0fa3d2c [id=0a703b25b63d4578b7eff94008a90dea]) successfully deleted
I20251029 08:52:31.195087 1179 tablet_replica.cc:333] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.195107 1178 raft_consensus.cc:359] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.195330 1178 raft_consensus.cc:385] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.195400 1178 raft_consensus.cc:740] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.195369 1179 raft_consensus.cc:2243] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:31.195616 1178 consensus_queue.cc:260] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.195900 1179 pending_rounds.cc:70] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:31.195983 1179 pending_rounds.cc:77] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037682724147200 op_type: NO_OP noop_request { }
I20251029 08:52:31.196151 1179 raft_consensus.cc:2889] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:31.196187 1178 ts_tablet_manager.cc:1434] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:31.196314 1179 raft_consensus.cc:2272] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.196475 1178 tablet_bootstrap.cc:492] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.197019 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_079d27f45f6a49e093d99a073678a387" } modify_external_catalogs: true
I20251029 08:52:31.197165 1183 tablet_bootstrap.cc:492] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:31.197185 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_079d27f45f6a49e093d99a073678a387" } modify_external_catalogs: true
I20251029 08:52:31.197276 1183 ts_tablet_manager.cc:1403] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.302s user 0.008s sys 0.001s
I20251029 08:52:31.198016 1183 raft_consensus.cc:359] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.198159 1183 raft_consensus.cc:385] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.198189 1178 tablet_bootstrap.cc:654] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.198211 1183 raft_consensus.cc:740] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.197479 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_66e64babc1194427be120879d68f8617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.199080 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66e64babc1194427be120879d68f8617 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.199625 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 59c90087c9e94665a9b3fac798304af7 (table default.Table_645b4972920f4607880280733ebe04a6 [id=5b45ec162f5e43d4a7976bb7f6426dbd]). Expected version 1 got 0
W20251029 08:52:31.205525 996 consensus_peers.cc:597] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.213200 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 5ad98112c0ea4f59980e72165cef8a1f
I20251029 08:52:31.214078 1178 tablet_bootstrap.cc:492] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.214221 1178 ts_tablet_manager.cc:1403] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20251029 08:52:31.214996 1178 raft_consensus.cc:359] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.215189 1178 raft_consensus.cc:385] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.215245 1178 raft_consensus.cc:740] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.215440 1178 consensus_queue.cc:260] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.215927 1178 ts_tablet_manager.cc:1434] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:31.216204 1178 tablet_bootstrap.cc:492] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.217473 1183 consensus_queue.cc:260] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.217916 1178 tablet_bootstrap.cc:654] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.218000 1183 ts_tablet_manager.cc:1434] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251029 08:52:31.218274 1183 tablet_bootstrap.cc:492] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:31.218433 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.220183 1183 tablet_bootstrap.cc:654] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.229095 1178 tablet_bootstrap.cc:492] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.229245 1178 ts_tablet_manager.cc:1403] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20251029 08:52:31.230022 1178 raft_consensus.cc:359] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.230207 1178 raft_consensus.cc:385] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.230255 1178 raft_consensus.cc:740] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.230481 1178 consensus_queue.cc:260] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.231012 1178 ts_tablet_manager.cc:1434] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:31.231293 1178 tablet_bootstrap.cc:492] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.232517 745 tablet_service.cc:1505] Processing CreateTablet for tablet 4675f0aec6a14807afd3094780d60c72 (DEFAULT_TABLE table=default.Table_d2361c16d6164246a7b81385f8fcf1c6 [id=28b9344d4f194f42a186c230ea3153f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.233273 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 59c90087c9e94665a9b3fac798304af7 (table default.Table_645b4972920f4607880280733ebe04a6 [id=5b45ec162f5e43d4a7976bb7f6426dbd]). Expected version 1 got 0
I20251029 08:52:31.233596 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa]). Expected version 2 got 0
I20251029 08:52:31.240262 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 442c37e1ee724b6b96f679f05640baf9 (table default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb]). Expected version 1 got 0
I20251029 08:52:31.241065 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 57159c264d6641afa3415d33c1d32efe (table default.Table_80492c8792f54daaa09fb96f4fa0b0f5 [id=7a5c0a7295414daa823ea0df7d47764d]). Expected version 1 got 0
I20251029 08:52:31.242177 2797 raft_consensus.cc:493] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.242316 2797 raft_consensus.cc:515] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.242887 2797 leader_election.cc:290] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.243528 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:31.243595 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.243741 778 raft_consensus.cc:2468] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:31.244242 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:31.244750 998 leader_election.cc:304] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:31.245314 2797 raft_consensus.cc:2804] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:31.245386 2797 raft_consensus.cc:493] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:31.245445 2797 raft_consensus.cc:3060] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:31.245393 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.246073 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 62597189ecb047788b83788b514e9099 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:31.248349 996 leader_election.cc:343] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.248756 2797 raft_consensus.cc:515] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.249745 2797 leader_election.cc:290] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.249873 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:31.250097 778 raft_consensus.cc:3060] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:31.250703 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dec33b502aa4916b89b878462de1ab1" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:31.251226 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 99455.3us
W20251029 08:52:31.251259 996 leader_election.cc:343] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.251456 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ad98112c0ea4f59980e72165cef8a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:31.252218 861 leader_election.cc:343] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 43c3074caee74069a8a7f1b87721d726
I20251029 08:52:31.252367 861 leader_election.cc:304] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:31.252629 2731 raft_consensus.cc:2749] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 08:52:31.253177 778 raft_consensus.cc:2468] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:31.253620 998 leader_election.cc:304] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.254045 2797 raft_consensus.cc:2804] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:31.254510 2797 raft_consensus.cc:697] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:31.254822 2797 consensus_queue.cc:237] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.259186 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.261548 1179 ts_tablet_manager.cc:1916] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.264140 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6eb0f7c99526409c86c08f5dfbe63e58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:31.264832 1177 raft_consensus.cc:385] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.264973 1177 raft_consensus.cc:740] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.265105 764 tablet_service.cc:1552] Processing DeleteTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.265303 1177 consensus_queue.cc:260] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.266049 1177 ts_tablet_manager.cc:1434] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.077s user 0.001s sys 0.000s
I20251029 08:52:31.263429 2707 raft_consensus.cc:493] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.269361 2707 raft_consensus.cc:515] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.270007 2707 leader_election.cc:290] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.270663 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:31.270862 778 raft_consensus.cc:2468] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:31.271255 998 leader_election.cc:304] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:31.271507 2707 raft_consensus.cc:2804] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:31.271574 2707 raft_consensus.cc:493] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:31.271632 2707 raft_consensus.cc:3060] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:31.274297 2707 raft_consensus.cc:515] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.274875 2707 leader_election.cc:290] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.275647 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:31.275816 778 raft_consensus.cc:3060] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:31.276189 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb0f7c99526409c86c08f5dfbe63e58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.278249 592 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa]). Expected version 2 got 0
I20251029 08:52:31.278882 778 raft_consensus.cc:2468] T 66329e1e548949cc9d59097953288375 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:31.279400 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 34729c3f97a44713a352bde297a80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.280475 998 leader_election.cc:304] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:31.280695 2359 raft_consensus.cc:493] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.280800 2359 raft_consensus.cc:515] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.281296 2359 leader_election.cc:290] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.281415 2359 raft_consensus.cc:2804] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:31.281493 2359 raft_consensus.cc:697] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:31.281710 2359 consensus_queue.cc:237] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.287983 1178 tablet_bootstrap.cc:654] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.287992 1183 tablet_bootstrap.cc:492] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:31.288158 1183 ts_tablet_manager.cc:1403] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.006s
I20251029 08:52:31.288919 1183 raft_consensus.cc:359] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.289098 1183 raft_consensus.cc:385] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.289165 1183 raft_consensus.cc:740] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.289366 1183 consensus_queue.cc:260] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.289889 1183 ts_tablet_manager.cc:1434] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:31.290175 1183 tablet_bootstrap.cc:492] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:31.291612 1183 tablet_bootstrap.cc:654] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.292419 2895 tablet.cc:1722] T ec54e736a3e64dd7a839252c440f778c P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 08:52:31.292960 998 leader_election.cc:343] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 663d6587a5e14519afbb0f94d691d1b4
I20251029 08:52:31.297335 2359 raft_consensus.cc:493] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.297519 2359 raft_consensus.cc:515] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.298723 2359 leader_election.cc:290] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.298806 1178 tablet_bootstrap.cc:492] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.298943 1178 ts_tablet_manager.cc:1403] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.068s user 0.020s sys 0.000s
I20251029 08:52:31.299590 1178 raft_consensus.cc:359] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.299754 1178 raft_consensus.cc:385] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.299809 1178 raft_consensus.cc:740] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.300037 1178 consensus_queue.cc:260] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.300510 1178 ts_tablet_manager.cc:1434] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251029 08:52:31.300770 1178 tablet_bootstrap.cc:492] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.302177 1178 tablet_bootstrap.cc:654] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.305099 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4675f0aec6a14807afd3094780d60c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.309619 1179 ts_tablet_manager.cc:1929] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:31.312158 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:31.312355 774 raft_consensus.cc:2468] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
W20251029 08:52:31.316062 582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38825, user_credentials={real_user=slave}} blocked reactor thread for 52246.1us
I20251029 08:52:31.317135 1178 tablet_bootstrap.cc:492] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.317265 1178 ts_tablet_manager.cc:1403] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20251029 08:52:31.270437 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.275229 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:31.317729 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 101aff0682d747dca3846bf95bce0a5d consensus state has no leader
I20251029 08:52:31.280905 2889 tablet.cc:1722] T ec54e736a3e64dd7a839252c440f778c P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:31.281773 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "663d6587a5e14519afbb0f94d691d1b4" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.317948 1178 raft_consensus.cc:359] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.318095 1178 raft_consensus.cc:385] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.318141 1178 raft_consensus.cc:740] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.298784 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.318332 1178 consensus_queue.cc:260] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.318801 1178 ts_tablet_manager.cc:1434] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:31.319056 1178 tablet_bootstrap.cc:492] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.320236 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 58932f38c0d64aec9a02238eaea56a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.324077 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6944e94b47b34496aa62902b739cfa85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.324599 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6944e94b47b34496aa62902b739cfa85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.325615 1178 tablet_bootstrap.cc:654] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.312747 998 leader_election.cc:304] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:31.326289 1179 log.cc:1199] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/7046bb1fcdd34d308aecddfdf0752911
I20251029 08:52:31.326483 2359 raft_consensus.cc:2804] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:31.326560 2359 raft_consensus.cc:493] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:31.326619 2359 raft_consensus.cc:3060] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:31.326648 1179 ts_tablet_manager.cc:1950] T 7046bb1fcdd34d308aecddfdf0752911 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.327390 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1188) took 1529 ms. Trace:
I20251029 08:52:31.327463 1179 rpcz_store.cc:276] 1029 08:52:29.798415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.799306 (+ 891us) service_pool.cc:225] Handling call
1029 08:52:31.195034 (+1395728us) ts_tablet_manager.cc:1151] Deleting tablet 7046bb1fcdd34d308aecddfdf0752911
1029 08:52:31.195045 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.306436 (+111391us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.309532 (+ 3096us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.327371 (+ 17839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395586}
I20251029 08:52:31.327786 1179 tablet_replica.cc:333] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.327980 1179 raft_consensus.cc:2243] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.328176 1179 raft_consensus.cc:2272] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.328711 1179 ts_tablet_manager.cc:1916] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.329278 2359 raft_consensus.cc:515] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:31.329778 996 leader_election.cc:343] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.329867 2359 leader_election.cc:290] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:31.330080 996 leader_election.cc:343] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.325091 1183 tablet_bootstrap.cc:492] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
W20251029 08:52:31.330332 996 leader_election.cc:343] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.330381 1183 ts_tablet_manager.cc:1403] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20251029 08:52:31.330610 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 7046bb1fcdd34d308aecddfdf0752911 (table default.Table_e9b105d36aed4a0da8586038f56dec12 [id=463d151caf764effa17b75b7875c5e6f]) successfully deleted
I20251029 08:52:31.330412 996 leader_election.cc:304] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.331058 2359 raft_consensus.cc:2749] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.331364 996 leader_election.cc:343] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.332994 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b3c1dbf9db424137b9c0cec53bb32311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.333542 596 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3c1dbf9db424137b9c0cec53bb32311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.337417 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:31.337998 996 leader_election.cc:343] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.338829 2869 raft_consensus.cc:493] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.338963 2869 raft_consensus.cc:515] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.339890 2869 leader_election.cc:290] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.342996 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4200db822deb4ace936f561c230ac1fb" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:31.343168 774 raft_consensus.cc:3060] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
W20251029 08:52:31.344264 589 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 491fb831cb2d4dbe9908fb6a571a43ca consensus state has no leader
I20251029 08:52:31.344595 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_ad961d1397114acab3368b402e9fd83f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.345135 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad961d1397114acab3368b402e9fd83f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.346041 597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet e1e2be77182e4259b76d20cae4e9e478
I20251029 08:52:31.346057 774 raft_consensus.cc:2468] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:31.346458 597 catalog_manager.cc:6719] ReplaceTablet: tablet e1e2be77182e4259b76d20cae4e9e478 deleted and replaced by tablet d8df074547624d449c46dba235bf7d7a
I20251029 08:52:31.346513 998 leader_election.cc:304] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.346764 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_c7cec907d0c54aa297ab2041445ef20e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:31.346959 2359 raft_consensus.cc:2804] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
W20251029 08:52:31.347283 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7cec907d0c54aa297ab2041445ef20e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.347416 2359 raft_consensus.cc:697] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:31.347698 2359 consensus_queue.cc:237] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.348129 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_b902ca54a7614b7086c649649b3a6646"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.348568 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b902ca54a7614b7086c649649b3a6646 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.350045 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8970ca1f227849a1a99ac5d33e450e14
I20251029 08:52:31.350687 591 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_40f4c90bb63e4ebdbf7af5f31149e57e" } modify_external_catalogs: true
I20251029 08:52:31.350826 591 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_40f4c90bb63e4ebdbf7af5f31149e57e" } modify_external_catalogs: true
I20251029 08:52:31.353868 764 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e2be77182e4259b76d20cae4e9e478 with delete_type TABLET_DATA_DELETED (replaced by tablet d8df074547624d449c46dba235bf7d7a) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.354082 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.355898 1179 ts_tablet_manager.cc:1929] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.357537 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 8a85a1dff3034284a6dcf0e8c3f53297
I20251029 08:52:31.358565 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.359059 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.360034 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_98d653f845be48e7bb3ccd4d1000e926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.361404 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98d653f845be48e7bb3ccd4d1000e926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.362133 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d444a5d489434696ae178c6d650bf8d5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.362643 718 leader_election.cc:343] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.356956 1047 tablet_service.cc:1552] Processing DeleteTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.368698 1179 log.cc:1199] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/65b20d53d6214bdc81ca205dbf922d7b
I20251029 08:52:31.369068 1179 ts_tablet_manager.cc:1950] T 65b20d53d6214bdc81ca205dbf922d7b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.369899 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1191) took 1570 ms. Trace:
I20251029 08:52:31.369992 1179 rpcz_store.cc:276] 1029 08:52:29.799656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.799701 (+ 45us) service_pool.cc:225] Handling call
1029 08:52:31.327736 (+1528035us) ts_tablet_manager.cc:1151] Deleting tablet 65b20d53d6214bdc81ca205dbf922d7b
1029 08:52:31.327747 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.351881 (+ 24134us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.355793 (+ 3912us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.369877 (+ 14084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527878}
I20251029 08:52:31.370404 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1198) took 1436 ms. Trace:
I20251029 08:52:31.357172 1051 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e2be77182e4259b76d20cae4e9e478 with delete_type TABLET_DATA_DELETED (replaced by tablet d8df074547624d449c46dba235bf7d7a) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.370527 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 65b20d53d6214bdc81ca205dbf922d7b (table default.Table_9c4c2744d70f4d1a9e52b9604edcb436 [id=5762d46725ec489d92cc4cb914fe070b]) successfully deleted
I20251029 08:52:31.357278 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d444a5d489434696ae178c6d650bf8d5" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.362318 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a85a1dff3034284a6dcf0e8c3f53297 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:30 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.331341 1183 raft_consensus.cc:359] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:31.371049 720 leader_election.cc:343] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.371196 1183 raft_consensus.cc:385] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.371186 720 leader_election.cc:304] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.371400 897 tablet_service.cc:1552] Processing DeleteTablet for tablet 8970ca1f227849a1a99ac5d33e450e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.371587 1183 raft_consensus.cc:740] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.371788 2869 raft_consensus.cc:2749] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.372380 1183 consensus_queue.cc:260] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.373791 1183 ts_tablet_manager.cc:1434] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20251029 08:52:31.374725 1183 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:31.374974 893 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e2be77182e4259b76d20cae4e9e478 with delete_type TABLET_DATA_DELETED (replaced by tablet d8df074547624d449c46dba235bf7d7a) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.377559 1176 ts_tablet_manager.cc:1929] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.377681 1176 log.cc:1199] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b1aaf233eba2461f9182e094a0115a4e
I20251029 08:52:31.378031 1176 ts_tablet_manager.cc:1950] T b1aaf233eba2461f9182e094a0115a4e P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:31.378834 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1176) took 1586 ms. Trace:
I20251029 08:52:31.378945 1176 rpcz_store.cc:276] 1029 08:52:29.792601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.792674 (+ 73us) service_pool.cc:225] Handling call
1029 08:52:31.056086 (+1263412us) ts_tablet_manager.cc:1151] Deleting tablet b1aaf233eba2461f9182e094a0115a4e
1029 08:52:31.056095 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.304694 (+248599us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.377427 (+ 72733us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.378806 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244444}
I20251029 08:52:31.379369 1176 tablet_replica.cc:333] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:31.379570 1176 raft_consensus.cc:2243] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.379601 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b1aaf233eba2461f9182e094a0115a4e (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]) successfully deleted
I20251029 08:52:31.379712 1176 raft_consensus.cc:2272] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.382382 1974 ts_tablet_manager.cc:1929] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:31.382514 1974 log.cc:1199] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/5f5ebd2bf6614980ac0c18f5d9974fc0
I20251029 08:52:31.382948 1974 ts_tablet_manager.cc:1950] T 5f5ebd2bf6614980ac0c18f5d9974fc0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:31.383730 1183 tablet_bootstrap.cc:654] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.395231 1178 tablet_bootstrap.cc:492] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.395486 1178 ts_tablet_manager.cc:1403] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.020s
I20251029 08:52:31.396245 1178 raft_consensus.cc:359] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.396615 1178 raft_consensus.cc:385] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.396718 1178 raft_consensus.cc:740] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.396976 1178 consensus_queue.cc:260] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.397505 1178 ts_tablet_manager.cc:1434] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:31.397841 1178 tablet_bootstrap.cc:492] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.402199 1178 tablet_bootstrap.cc:654] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.403069 1183 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:31.403223 1183 ts_tablet_manager.cc:1403] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251029 08:52:31.404126 1183 raft_consensus.cc:359] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.404448 1183 raft_consensus.cc:385] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 08:52:31.407740 861 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 09968d93203140fa9201928c87702ee0
W20251029 08:52:31.408077 861 consensus_peers.cc:597] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3955ff79faf42bb82f764aae79b85d9. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:31.408761 861 consensus_peers.cc:597] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9bbf6401423411089c9070f3b03f1a7. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.404520 1183 raft_consensus.cc:740] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.412011 1183 consensus_queue.cc:260] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.412578 1183 ts_tablet_manager.cc:1434] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 08:52:31.413017 1183 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:31.413061 861 leader_election.cc:343] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 657cdaad97e24611a997e26a6293ea5e
W20251029 08:52:31.413424 861 consensus_peers.cc:597] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cc95257c937454aa6cd55b5a9394a56. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:31.413828 861 leader_election.cc:343] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 09968d93203140fa9201928c87702ee0
I20251029 08:52:31.413997 861 leader_election.cc:304] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:31.414386 2731 raft_consensus.cc:2749] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.414970 1183 tablet_bootstrap.cc:654] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.425794 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 442c37e1ee724b6b96f679f05640baf9 (table default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb]). Expected version 1 got 0
I20251029 08:52:31.426069 592 catalog_manager.cc:5649] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:31.426466 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:31.432049 594 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_fa532317de53450b9330eedb3260b1e8 [id=f13f5c5b7d7742bf9d4a28f2513ec906] alter complete (version 1)
I20251029 08:52:31.435047 594 master_service.cc:954] ReplaceTablet: received request to replace tablet adb59d3ee338465a942e6b4423f0e071 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:31.438868 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb380018f46465e82ac6223ecf8dd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:09 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.443325 750 tablet_service.cc:1505] Processing CreateTablet for tablet a3631548b36644528bde2c3a2614c44c (DEFAULT_TABLE table=default.Table_de20d4a8539d454fab53870dc689bc48 [id=829e4bc39eae4e539a9f92aa8e458daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.443750 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3631548b36644528bde2c3a2614c44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.449311 897 tablet_service.cc:1505] Processing CreateTablet for tablet ecf510abf5b6496c87e586592c32c3af (DEFAULT_TABLE table=default.Table_bab242baf18b4ff8a3cb134178846f31 [id=939145a7122e42bba8cbab4ca7270b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.449685 895 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7774c358f14e1687043ee3b8cd6ded (DEFAULT_TABLE table=default.Table_25119faaae7141a8b57594132e159677 [id=f1298af8281142f3a300dc505ab19940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.450487 893 tablet_service.cc:1505] Processing CreateTablet for tablet a3631548b36644528bde2c3a2614c44c (DEFAULT_TABLE table=default.Table_de20d4a8539d454fab53870dc689bc48 [id=829e4bc39eae4e539a9f92aa8e458daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.452155 753 tablet_service.cc:1505] Processing CreateTablet for tablet 70732c0cfe6a4478a8fe329312781234 (DEFAULT_TABLE table=default.Table_ac317ca8b45f460f88ecb16a1bf0d802 [id=8c6933dc05d9469c9a884a9b39486c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.452349 746 tablet_service.cc:1505] Processing CreateTablet for tablet f54e0b4b332b43f7a0a4ccf01d7816d4 (DEFAULT_TABLE table=default.Table_0c8626ad4f874c99842ee20fc62cf6c0 [id=55b5d914a0e94771b8360fa8b3ff9068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.452564 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70732c0cfe6a4478a8fe329312781234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.452510 764 tablet_service.cc:1505] Processing CreateTablet for tablet ecf510abf5b6496c87e586592c32c3af (DEFAULT_TABLE table=default.Table_bab242baf18b4ff8a3cb134178846f31 [id=939145a7122e42bba8cbab4ca7270b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.452780 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54e0b4b332b43f7a0a4ccf01d7816d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.452944 758 tablet_service.cc:1505] Processing CreateTablet for tablet 90fe71e31dbf42288f66f4160a13ed90 (DEFAULT_TABLE table=default.Table_b2024dec46a943a8a1e4866ada0a388e [id=a880935fa64c4f38bc47f0f1a67d3baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.453233 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fe71e31dbf42288f66f4160a13ed90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.456260 749 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7774c358f14e1687043ee3b8cd6ded (DEFAULT_TABLE table=default.Table_25119faaae7141a8b57594132e159677 [id=f1298af8281142f3a300dc505ab19940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.456574 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7774c358f14e1687043ee3b8cd6ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.459678 1178 tablet_bootstrap.cc:492] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.459807 1178 ts_tablet_manager.cc:1403] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.062s user 0.018s sys 0.024s
I20251029 08:52:31.370483 1179 rpcz_store.cc:276] 1029 08:52:29.933946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.934007 (+ 61us) service_pool.cc:225] Handling call
1029 08:52:31.370273 (+1436266us) ts_tablet_manager.cc:1151] Deleting tablet 307262da8888461ebddaaf33a31be8b8
1029 08:52:31.370284 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.370388 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436100}
I20251029 08:52:31.460266 747 tablet_service.cc:1505] Processing CreateTablet for tablet 881c0747d52d46c680b295fc93916d6c (DEFAULT_TABLE table=default.Table_b1be633f29d54c23b4a29d851377d3ec [id=f199514b01064406a7574c5780fc97a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.460327 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1199) took 1523 ms. Trace:
I20251029 08:52:31.460407 1179 rpcz_store.cc:276] 1029 08:52:29.936363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.936422 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:31.460179 (+1523757us) ts_tablet_manager.cc:1151] Deleting tablet 4675f0aec6a14807afd3094780d60c72
1029 08:52:31.460191 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.460306 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1523572}
I20251029 08:52:31.460580 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881c0747d52d46c680b295fc93916d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.460777 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1197) took 1528 ms. Trace:
I20251029 08:52:31.460847 1179 rpcz_store.cc:276] 1029 08:52:29.931948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.953782 (+ 21834us) service_pool.cc:225] Handling call
1029 08:52:31.460672 (+1506890us) ts_tablet_manager.cc:1151] Deleting tablet ea40ed38c0ba4d2f9045188af226ced5
1029 08:52:31.460680 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.460762 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506724}
I20251029 08:52:31.461135 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1200) took 1508 ms. Trace:
I20251029 08:52:31.461194 1179 rpcz_store.cc:276] 1029 08:52:29.953173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.953978 (+ 805us) service_pool.cc:225] Handling call
1029 08:52:31.461045 (+1507067us) ts_tablet_manager.cc:1151] Deleting tablet 4524a248859448c5b789c7ae82435705
1029 08:52:31.461053 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.461122 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506952}
I20251029 08:52:31.462252 1178 raft_consensus.cc:359] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.462414 1178 raft_consensus.cc:385] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.462467 1178 raft_consensus.cc:740] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
W20251029 08:52:31.462541 861 leader_election.cc:343] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 657cdaad97e24611a997e26a6293ea5e
I20251029 08:52:31.462699 861 leader_election.cc:304] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:31.462671 1178 consensus_queue.cc:260] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.463022 2731 raft_consensus.cc:2749] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.463163 1178 ts_tablet_manager.cc:1434] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:31.463414 1178 tablet_bootstrap.cc:492] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.464547 1176 ts_tablet_manager.cc:1916] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.464926 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf510abf5b6496c87e586592c32c3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.466629 1179 tablet_replica.cc:333] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.466823 1179 raft_consensus.cc:2243] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.466979 1179 raft_consensus.cc:2272] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.467492 1179 ts_tablet_manager.cc:1916] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.467738 2707 raft_consensus.cc:493] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.467865 2707 raft_consensus.cc:515] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.468457 2707 leader_election.cc:290] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.469516 591 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 39578ce0411646eea7f6d8dc61490e03
I20251029 08:52:31.470906 1176 ts_tablet_manager.cc:1929] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.471002 1176 log.cc:1199] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d306be1d51674c2abc506d248e05a9c7
I20251029 08:52:31.471278 1176 ts_tablet_manager.cc:1950] T d306be1d51674c2abc506d248e05a9c7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:31.473326 904 tablet_service.cc:1505] Processing CreateTablet for tablet 881c0747d52d46c680b295fc93916d6c (DEFAULT_TABLE table=default.Table_b1be633f29d54c23b4a29d851377d3ec [id=f199514b01064406a7574c5780fc97a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.473766 889 tablet_service.cc:1505] Processing CreateTablet for tablet ff85d521ec9f4f09b73ca0ea6541cd9e (DEFAULT_TABLE table=default.Table_702694262c424e65a9c1319f5c82eba3 [id=824896f76b5946caa32aa9f73b78e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.474098 894 tablet_service.cc:1505] Processing CreateTablet for tablet 02c601d8c3434b02ac480be7f59cc60b (DEFAULT_TABLE table=default.Table_cdaa068607404571bc2534b6ad0a3b6e [id=85404d664a614fc386efc15a2213bec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.474642 899 tablet_service.cc:1505] Processing CreateTablet for tablet 263a2a981cb44c04bc881f8656338760 (DEFAULT_TABLE table=default.Table_f330ad2e9ac74e1c8ec0b43d0405a238 [id=5f831078894740c6a68f0191a9fdfa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.474977 902 tablet_service.cc:1505] Processing CreateTablet for tablet 35db4974b5b94e1ea0df16a086f90ac5 (DEFAULT_TABLE table=default.Table_d101fe184fa848c19ec0ca5cc80ae445 [id=26e2556ffbab414a81c91e6381074343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.475273 898 tablet_service.cc:1505] Processing CreateTablet for tablet 2343d583cd8a4e04964316f256e22b6d (DEFAULT_TABLE table=default.Table_3edf34ad561b40ef83f837d36eb91335 [id=1563514ebd244d47a74c990e34f95866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.475487 1179 ts_tablet_manager.cc:1929] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.475589 1179 log.cc:1199] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3cd2e9a95b6c40be9e17397a313d249b
I20251029 08:52:31.475570 892 tablet_service.cc:1505] Processing CreateTablet for tablet a1f138b5e5a34239843fff6b69fc43b3 (DEFAULT_TABLE table=default.Table_697b189fcd394daa8cb0fdb8d0c87518 [id=58130a98d1d44fddbf4f4c949c450987]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.475904 1179 ts_tablet_manager.cc:1950] T 3cd2e9a95b6c40be9e17397a313d249b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.475917 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4644312dee44fd87047fb78e24d344 (DEFAULT_TABLE table=default.Table_48c90ff8955448b891db2bf3482a489c [id=4feb3504b35b4d268ba79635a70f057d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.476235 900 tablet_service.cc:1505] Processing CreateTablet for tablet 66a24dac06544e509eeb4a1c6072ba11 (DEFAULT_TABLE table=default.Table_d39067d9fd47411f95a64ff74c3c4df0 [id=2953d27984154faabc410182c8aaeac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.476544 890 tablet_service.cc:1505] Processing CreateTablet for tablet f11d8baf25334120a4a0327df5dff2d2 (DEFAULT_TABLE table=default.Table_93f45fc099dd4110a562e1fc85e8045c [id=8c4cf3e9900c493bb06870ccc5261950]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.476711 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1201) took 1522 ms. Trace:
I20251029 08:52:31.476830 903 tablet_service.cc:1505] Processing CreateTablet for tablet d010353b251140e99ce2df996f965991 (DEFAULT_TABLE table=default.Table_bb0e63a0c67b4931bc2282a6d83b4db1 [id=afc3cca2160c49cda9cd59abb5118a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.476795 1179 rpcz_store.cc:276] 1029 08:52:29.954691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.957195 (+ 2504us) service_pool.cc:225] Handling call
1029 08:52:31.461357 (+1504162us) ts_tablet_manager.cc:1151] Deleting tablet 3cd2e9a95b6c40be9e17397a313d249b
1029 08:52:31.464224 (+ 2867us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.472376 (+ 8152us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.475395 (+ 3019us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.476682 (+ 1287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503957}
I20251029 08:52:31.477074 1179 tablet_replica.cc:333] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.477226 1179 raft_consensus.cc:2243] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:31.477425 1179 pending_rounds.cc:70] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:31.477456 1179 pending_rounds.cc:77] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037685818994688 op_type: NO_OP noop_request { }
I20251029 08:52:31.477545 1179 raft_consensus.cc:2889] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:31.477612 1179 raft_consensus.cc:2272] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.478322 1178 tablet_bootstrap.cc:654] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.479687 1046 tablet_service.cc:1505] Processing CreateTablet for tablet ecf510abf5b6496c87e586592c32c3af (DEFAULT_TABLE table=default.Table_bab242baf18b4ff8a3cb134178846f31 [id=939145a7122e42bba8cbab4ca7270b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.479753 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 90fe71e31dbf42288f66f4160a13ed90 (DEFAULT_TABLE table=default.Table_b2024dec46a943a8a1e4866ada0a388e [id=a880935fa64c4f38bc47f0f1a67d3baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.480067 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf510abf5b6496c87e586592c32c3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.480281 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7774c358f14e1687043ee3b8cd6ded (DEFAULT_TABLE table=default.Table_25119faaae7141a8b57594132e159677 [id=f1298af8281142f3a300dc505ab19940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.480685 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 881c0747d52d46c680b295fc93916d6c (DEFAULT_TABLE table=default.Table_b1be633f29d54c23b4a29d851377d3ec [id=f199514b01064406a7574c5780fc97a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.481040 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fe71e31dbf42288f66f4160a13ed90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.483290 1179 ts_tablet_manager.cc:1916] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.483654 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 70732c0cfe6a4478a8fe329312781234 (DEFAULT_TABLE table=default.Table_ac317ca8b45f460f88ecb16a1bf0d802 [id=8c6933dc05d9469c9a884a9b39486c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.484074 1045 tablet_service.cc:1505] Processing CreateTablet for tablet f54e0b4b332b43f7a0a4ccf01d7816d4 (DEFAULT_TABLE table=default.Table_0c8626ad4f874c99842ee20fc62cf6c0 [id=55b5d914a0e94771b8360fa8b3ff9068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.484397 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70732c0cfe6a4478a8fe329312781234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.484658 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54e0b4b332b43f7a0a4ccf01d7816d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.484969 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881c0747d52d46c680b295fc93916d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.488183 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7774c358f14e1687043ee3b8cd6ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.494853 1179 ts_tablet_manager.cc:1929] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:31.496732 1179 log.cc:1199] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b1aaf233eba2461f9182e094a0115a4e
W20251029 08:52:31.497821 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 307262da8888461ebddaaf33a31be8b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 307262da8888461ebddaaf33a31be8b8 already in progress: creating tablet
I20251029 08:52:31.498016 1179 ts_tablet_manager.cc:1950] T b1aaf233eba2461f9182e094a0115a4e P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:31.498186 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4675f0aec6a14807afd3094780d60c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4675f0aec6a14807afd3094780d60c72 already in progress: creating tablet
W20251029 08:52:31.498355 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ea40ed38c0ba4d2f9045188af226ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea40ed38c0ba4d2f9045188af226ced5 already in progress: creating tablet
W20251029 08:52:31.498509 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4524a248859448c5b789c7ae82435705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4524a248859448c5b789c7ae82435705 already in progress: creating tablet
I20251029 08:52:31.498647 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3cd2e9a95b6c40be9e17397a313d249b (table default.Table_79896e97f2934c50a068bf4beaf34966 [id=c6901c2dab494740be7681afa3f4aa9f]) successfully deleted
I20251029 08:52:31.499379 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1202) took 1514 ms. Trace:
I20251029 08:52:31.500154 1179 rpcz_store.cc:276] 1029 08:52:29.984799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.985204 (+ 405us) service_pool.cc:225] Handling call
1029 08:52:31.477032 (+1491828us) ts_tablet_manager.cc:1151] Deleting tablet b1aaf233eba2461f9182e094a0115a4e
1029 08:52:31.477040 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.487606 (+ 10566us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.494756 (+ 7150us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.499359 (+ 4603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482494}
I20251029 08:52:31.500934 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1204) took 1483 ms. Trace:
I20251029 08:52:31.501534 1179 rpcz_store.cc:276] 1029 08:52:30.017597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.017649 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:31.500800 (+1483151us) ts_tablet_manager.cc:1151] Deleting tablet f2c18e4c41714ba38bdd2e208a5e761e
1029 08:52:31.500810 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.500916 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482923}
I20251029 08:52:31.501777 891 tablet_service.cc:1505] Processing CreateTablet for tablet 90fe71e31dbf42288f66f4160a13ed90 (DEFAULT_TABLE table=default.Table_b2024dec46a943a8a1e4866ada0a388e [id=a880935fa64c4f38bc47f0f1a67d3baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.502205 1179 tablet_replica.cc:333] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.502750 746 tablet_service.cc:1505] Processing CreateTablet for tablet 02c601d8c3434b02ac480be7f59cc60b (DEFAULT_TABLE table=default.Table_cdaa068607404571bc2534b6ad0a3b6e [id=85404d664a614fc386efc15a2213bec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.503001 754 tablet_service.cc:1505] Processing CreateTablet for tablet a1f138b5e5a34239843fff6b69fc43b3 (DEFAULT_TABLE table=default.Table_697b189fcd394daa8cb0fdb8d0c87518 [id=58130a98d1d44fddbf4f4c949c450987]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.503175 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c601d8c3434b02ac480be7f59cc60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.503446 760 tablet_service.cc:1505] Processing CreateTablet for tablet 66a24dac06544e509eeb4a1c6072ba11 (DEFAULT_TABLE table=default.Table_d39067d9fd47411f95a64ff74c3c4df0 [id=2953d27984154faabc410182c8aaeac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.503767 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a24dac06544e509eeb4a1c6072ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.508260 757 tablet_service.cc:1505] Processing CreateTablet for tablet 263a2a981cb44c04bc881f8656338760 (DEFAULT_TABLE table=default.Table_f330ad2e9ac74e1c8ec0b43d0405a238 [id=5f831078894740c6a68f0191a9fdfa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.508286 761 tablet_service.cc:1505] Processing CreateTablet for tablet d010353b251140e99ce2df996f965991 (DEFAULT_TABLE table=default.Table_bb0e63a0c67b4931bc2282a6d83b4db1 [id=afc3cca2160c49cda9cd59abb5118a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.508478 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2343d583cd8a4e04964316f256e22b6d (DEFAULT_TABLE table=default.Table_3edf34ad561b40ef83f837d36eb91335 [id=1563514ebd244d47a74c990e34f95866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.508602 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263a2a981cb44c04bc881f8656338760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.508705 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 39578ce0411646eea7f6d8dc61490e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.508766 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2343d583cd8a4e04964316f256e22b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.509016 751 tablet_service.cc:1505] Processing CreateTablet for tablet 55710ce1ddd140d2a390cc5d89e8cfac (DEFAULT_TABLE table=default.Table_620d0574d4154375bbf3774c5ad37097 [id=b84b5834b26b4c3c8129cedb169c6b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.509279 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55710ce1ddd140d2a390cc5d89e8cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.510447 896 tablet_service.cc:1505] Processing CreateTablet for tablet f54e0b4b332b43f7a0a4ccf01d7816d4 (DEFAULT_TABLE table=default.Table_0c8626ad4f874c99842ee20fc62cf6c0 [id=55b5d914a0e94771b8360fa8b3ff9068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512245 756 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4644312dee44fd87047fb78e24d344 (DEFAULT_TABLE table=default.Table_48c90ff8955448b891db2bf3482a489c [id=4feb3504b35b4d268ba79635a70f057d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512413 762 tablet_service.cc:1505] Processing CreateTablet for tablet 59348e20825e4be69b86738e6f6b79ba (DEFAULT_TABLE table=default.Table_6d00dde368eb4b14bb49537997d2a49f [id=c0528b1715cb4f1b89c04db3f8eba6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512809 759 tablet_service.cc:1505] Processing CreateTablet for tablet 0d5ee0b03ce840c5ac78d83a1054511a (DEFAULT_TABLE table=default.Table_301169b3168b41b789340c0724a5afcd [id=dde2397e469d41819322a39aad089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512246 755 tablet_service.cc:1505] Processing CreateTablet for tablet 3e83c0c4e85b41718db318f12d4518ae (DEFAULT_TABLE table=default.Table_77007e3fc52446939ef29f8189facb05 [id=43a99affb7f6438887efc4b7d85e1e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512246 763 tablet_service.cc:1505] Processing CreateTablet for tablet f11d8baf25334120a4a0327df5dff2d2 (DEFAULT_TABLE table=default.Table_93f45fc099dd4110a562e1fc85e8045c [id=8c4cf3e9900c493bb06870ccc5261950]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513403 753 tablet_service.cc:1505] Processing CreateTablet for tablet ff85d521ec9f4f09b73ca0ea6541cd9e (DEFAULT_TABLE table=default.Table_702694262c424e65a9c1319f5c82eba3 [id=824896f76b5946caa32aa9f73b78e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.508286 752 tablet_service.cc:1505] Processing CreateTablet for tablet 35db4974b5b94e1ea0df16a086f90ac5 (DEFAULT_TABLE table=default.Table_d101fe184fa848c19ec0ca5cc80ae445 [id=26e2556ffbab414a81c91e6381074343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.515400 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f138b5e5a34239843fff6b69fc43b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.515844 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d010353b251140e99ce2df996f965991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.517483 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4644312dee44fd87047fb78e24d344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.517690 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35db4974b5b94e1ea0df16a086f90ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.517763 764 tablet_service.cc:1505] Processing CreateTablet for tablet f7acde70ec7f4998ab63a37d025d35cf (DEFAULT_TABLE table=default.Table_cddcc5d90ac4457fbbe4489988f438f7 [id=63551108a9694c11afa79b5eac707956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.517856 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ee0b03ce840c5ac78d83a1054511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.518036 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7acde70ec7f4998ab63a37d025d35cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.518268 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e83c0c4e85b41718db318f12d4518ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.503787 998 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0bd490da0b0e47fc965343c123a27b22
I20251029 08:52:31.508181 1051 tablet_service.cc:1505] Processing CreateTablet for tablet a3631548b36644528bde2c3a2614c44c (DEFAULT_TABLE table=default.Table_de20d4a8539d454fab53870dc689bc48 [id=829e4bc39eae4e539a9f92aa8e458daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.518639 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11d8baf25334120a4a0327df5dff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.518868 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3631548b36644528bde2c3a2614c44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.518970 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59348e20825e4be69b86738e6f6b79ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.522476 2869 raft_consensus.cc:493] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.522601 2869 raft_consensus.cc:515] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.523483 2869 leader_election.cc:290] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.509399 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4644312dee44fd87047fb78e24d344 (DEFAULT_TABLE table=default.Table_48c90ff8955448b891db2bf3482a489c [id=4feb3504b35b4d268ba79635a70f057d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.524358 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4644312dee44fd87047fb78e24d344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.512465 1039 tablet_service.cc:1505] Processing CreateTablet for tablet ff85d521ec9f4f09b73ca0ea6541cd9e (DEFAULT_TABLE table=default.Table_702694262c424e65a9c1319f5c82eba3 [id=824896f76b5946caa32aa9f73b78e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512703 1049 tablet_service.cc:1505] Processing CreateTablet for tablet a1f138b5e5a34239843fff6b69fc43b3 (DEFAULT_TABLE table=default.Table_697b189fcd394daa8cb0fdb8d0c87518 [id=58130a98d1d44fddbf4f4c949c450987]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.512786 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 02c601d8c3434b02ac480be7f59cc60b (DEFAULT_TABLE table=default.Table_cdaa068607404571bc2534b6ad0a3b6e [id=85404d664a614fc386efc15a2213bec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513079 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 263a2a981cb44c04bc881f8656338760 (DEFAULT_TABLE table=default.Table_f330ad2e9ac74e1c8ec0b43d0405a238 [id=5f831078894740c6a68f0191a9fdfa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513342 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 35db4974b5b94e1ea0df16a086f90ac5 (DEFAULT_TABLE table=default.Table_d101fe184fa848c19ec0ca5cc80ae445 [id=26e2556ffbab414a81c91e6381074343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513466 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 66a24dac06544e509eeb4a1c6072ba11 (DEFAULT_TABLE table=default.Table_d39067d9fd47411f95a64ff74c3c4df0 [id=2953d27984154faabc410182c8aaeac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.524852 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff85d521ec9f4f09b73ca0ea6541cd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.525077 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c601d8c3434b02ac480be7f59cc60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.513726 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 59348e20825e4be69b86738e6f6b79ba (DEFAULT_TABLE table=default.Table_6d00dde368eb4b14bb49537997d2a49f [id=c0528b1715cb4f1b89c04db3f8eba6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513926 1038 tablet_service.cc:1505] Processing CreateTablet for tablet f11d8baf25334120a4a0327df5dff2d2 (DEFAULT_TABLE table=default.Table_93f45fc099dd4110a562e1fc85e8045c [id=8c4cf3e9900c493bb06870ccc5261950]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.528208 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263a2a981cb44c04bc881f8656338760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.528426 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11d8baf25334120a4a0327df5dff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.514232 1040 tablet_service.cc:1505] Processing CreateTablet for tablet d010353b251140e99ce2df996f965991 (DEFAULT_TABLE table=default.Table_bb0e63a0c67b4931bc2282a6d83b4db1 [id=afc3cca2160c49cda9cd59abb5118a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.528647 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35db4974b5b94e1ea0df16a086f90ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.528884 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d010353b251140e99ce2df996f965991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.529002 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a24dac06544e509eeb4a1c6072ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.530920 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 0d5ee0b03ce840c5ac78d83a1054511a (DEFAULT_TABLE table=default.Table_301169b3168b41b789340c0724a5afcd [id=dde2397e469d41819322a39aad089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.531221 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ee0b03ce840c5ac78d83a1054511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.531342 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b1aaf233eba2461f9182e094a0115a4e (table default.Table_4f1be1b978b7433896bb423e059679b9 [id=c94dfe6f611a47818ae1cc4602cffd03]) successfully deleted
W20251029 08:52:31.531546 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f2c18e4c41714ba38bdd2e208a5e761e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c18e4c41714ba38bdd2e208a5e761e already in progress: creating tablet
I20251029 08:52:31.532176 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f138b5e5a34239843fff6b69fc43b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.514408 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 39578ce0411646eea7f6d8dc61490e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.533640 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 55710ce1ddd140d2a390cc5d89e8cfac (DEFAULT_TABLE table=default.Table_620d0574d4154375bbf3774c5ad37097 [id=b84b5834b26b4c3c8129cedb169c6b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.514657 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 3e83c0c4e85b41718db318f12d4518ae (DEFAULT_TABLE table=default.Table_77007e3fc52446939ef29f8189facb05 [id=43a99affb7f6438887efc4b7d85e1e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.513638 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 2343d583cd8a4e04964316f256e22b6d (DEFAULT_TABLE table=default.Table_3edf34ad561b40ef83f837d36eb91335 [id=1563514ebd244d47a74c990e34f95866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.533987 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59348e20825e4be69b86738e6f6b79ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.538847 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_306a00708d9448639fca68a831b3bef3" } new_table_name: "default.Table_973a14639b1047f8b2e74a6e4a25c3d2" modify_external_catalogs: true
I20251029 08:52:31.543720 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff85d521ec9f4f09b73ca0ea6541cd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.546550 596 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_20173eef43ed4e66acd14cb73a7d2ebd" } new_table_name: "default.Table_05bf3ddf74a24cd3abd74b8eab043b1a" modify_external_catalogs: true
I20251029 08:52:31.547972 595 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_c5f70a8f965d4999a64b666bffc151c1" } new_table_name: "default.Table_a7d94cf13cd648eaaddb49ebbbd78dc6" modify_external_catalogs: true
I20251029 08:52:31.549268 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43aa92e7fba74d91ad0b832e5f5d6592" } modify_external_catalogs: true
I20251029 08:52:31.549391 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_43aa92e7fba74d91ad0b832e5f5d6592" } modify_external_catalogs: true
I20251029 08:52:31.550043 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9499d0099d1c4edc843fb8d29ca3b1b6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.550654 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 85df43bf781c421688f163331de0eb2b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:31.551060 588 master_service.cc:954] ReplaceTablet: received request to replace tablet a9fe3e9a507441b8ba5953c0120289c3 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:31.551443 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5" } modify_external_catalogs: true
I20251029 08:52:31.551568 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_46ab8dfc8f0e497dbdbcff3e147ac9f5" } modify_external_catalogs: true
W20251029 08:52:31.552806 720 leader_election.cc:343] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.553676 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55710ce1ddd140d2a390cc5d89e8cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.556190 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_33ff6ee160ac45db94ce4a84c3d5ad61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.556764 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33ff6ee160ac45db94ce4a84c3d5ad61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:31.557237 861 consensus_peers.cc:597] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Couldn't send request to peer 730d0f5b7abf43a293b7df13d01ce4ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ab56045bd814d59831a98618c759af4. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.557394 1047 tablet_service.cc:1505] Processing CreateTablet for tablet f7acde70ec7f4998ab63a37d025d35cf (DEFAULT_TABLE table=default.Table_cddcc5d90ac4457fbbe4489988f438f7 [id=63551108a9694c11afa79b5eac707956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.559284 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_d1f52fb5254c46f399a8df1a241755a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:31.559984 2707 raft_consensus.cc:493] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.560330 2707 raft_consensus.cc:515] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.560932 2707 leader_election.cc:290] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.560272 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6" } modify_external_catalogs: true
I20251029 08:52:31.561844 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_0b02f4f9ae3d409aa7e86c5c36459cd6" } modify_external_catalogs: true
W20251029 08:52:31.563431 591 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1f52fb5254c46f399a8df1a241755a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.564214 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet d306be1d51674c2abc506d248e05a9c7 with delete_type TABLET_DATA_DELETED (replaced by tablet a08630cbc401467a9dde1ce1e198cef7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.564563 1179 raft_consensus.cc:2243] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.565057 1179 raft_consensus.cc:2272] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.567050 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e83c0c4e85b41718db318f12d4518ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.566991 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_3584ce3292564e359247b74ca750e4d9" } new_table_name: "default.Table_b7290937457c4e9aad0d89f247f68a34" modify_external_catalogs: true
I20251029 08:52:31.566540 1179 ts_tablet_manager.cc:1916] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.577114 1179 ts_tablet_manager.cc:1929] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.579010 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet adb59d3ee338465a942e6b4423f0e071
I20251029 08:52:31.579483 594 catalog_manager.cc:6719] ReplaceTablet: tablet adb59d3ee338465a942e6b4423f0e071 deleted and replaced by tablet afb32905315b4077926d028eaa404967
I20251029 08:52:31.580096 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_3ee1d43136654538bff6cb34c23ff8da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:31.580729 594 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ee1d43136654538bff6cb34c23ff8da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:31.581272 2707 raft_consensus.cc:493] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.581489 2707 raft_consensus.cc:515] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.582116 2707 leader_election.cc:290] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:31.582507 1179 log.cc:1199] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/dbf9e36c47be4509a998be8e7138475b
I20251029 08:52:31.584556 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet adb59d3ee338465a942e6b4423f0e071 with delete_type TABLET_DATA_DELETED (replaced by tablet afb32905315b4077926d028eaa404967) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:31.584641 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 760fcb61d5274487b11a760573dfca6a (DEFAULT_TABLE table=default.Table_5e9da1c94f3e437d9b2e61d9a51a18a8 [id=485f1a12bf564f58918e15156e15d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.584822 760 tablet_service.cc:1552] Processing DeleteTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.585040 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760fcb61d5274487b11a760573dfca6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.585165 905 tablet_service.cc:1505] Processing CreateTablet for tablet 70732c0cfe6a4478a8fe329312781234 (DEFAULT_TABLE table=default.Table_ac317ca8b45f460f88ecb16a1bf0d802 [id=8c6933dc05d9469c9a884a9b39486c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.590112 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d306be1d51674c2abc506d248e05a9c7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.590628 922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0bd490da0b0e47fc965343c123a27b22" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.596082 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9499d0099d1c4edc843fb8d29ca3b1b6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.596529 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c2072c16fce43278f1e32b9672187e7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.606868 996 leader_election.cc:343] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:31.607102 996 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.607185 996 leader_election.cc:304] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.607419 2707 raft_consensus.cc:2749] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.607604 996 leader_election.cc:343] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.608776 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 036558cc29514284a91866c0997ba012 (DEFAULT_TABLE table=default.Table_ecbebd3f5d9b42218bb0dd513c106df2 [id=a307fb6f1d0141fa8ab9211651832bcb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:31.609063 998 leader_election.cc:343] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0c2072c16fce43278f1e32b9672187e7
I20251029 08:52:31.609169 998 leader_election.cc:304] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.609375 2707 raft_consensus.cc:2749] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.609740 718 leader_election.cc:343] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.609869 718 leader_election.cc:304] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:31.609876 998 leader_election.cc:343] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d306be1d51674c2abc506d248e05a9c7
I20251029 08:52:31.609970 998 leader_election.cc:304] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.610222 2869 raft_consensus.cc:2749] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.610435 2707 raft_consensus.cc:2749] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.610659 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d8beb60b87e44b5a698f1750b0777da consensus state has no leader
I20251029 08:52:31.609385 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 036558cc29514284a91866c0997ba012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.611085 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2343d583cd8a4e04964316f256e22b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.612072 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 42de32d87ebb451090e0310e36e64277 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.612349 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7acde70ec7f4998ab63a37d025d35cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.612561 1177 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:31.614415 1177 tablet_bootstrap.cc:654] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.618314 2888 tablet.cc:1722] T 442c37e1ee724b6b96f679f05640baf9 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:31.620132 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1177) took 1826 ms. Trace:
I20251029 08:52:31.621093 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881c0747d52d46c680b295fc93916d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.620143 2895 tablet.cc:1722] T 442c37e1ee724b6b96f679f05640baf9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:31.620242 1176 rpcz_store.cc:276] 1029 08:52:29.793189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.793239 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:31.379314 (+1586075us) ts_tablet_manager.cc:1151] Deleting tablet d306be1d51674c2abc506d248e05a9c7
1029 08:52:31.379327 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.467997 (+ 88670us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.470825 (+ 2828us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.620105 (+149280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567334}
I20251029 08:52:31.624298 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 39578ce0411646eea7f6d8dc61490e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.624696 901 tablet_service.cc:1505] Processing CreateTablet for tablet 3e83c0c4e85b41718db318f12d4518ae (DEFAULT_TABLE table=default.Table_77007e3fc52446939ef29f8189facb05 [id=43a99affb7f6438887efc4b7d85e1e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.625051 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e83c0c4e85b41718db318f12d4518ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.628203 1176 tablet_replica.cc:333] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:31.628441 1176 raft_consensus.cc:2243] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.628623 1176 raft_consensus.cc:2272] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.628712 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1036) took 5956 ms. Trace:
I20251029 08:52:31.628814 1974 rpcz_store.cc:276] 1029 08:52:25.672413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.672453 (+ 40us) service_pool.cc:225] Handling call
1029 08:52:30.852550 (+5180097us) ts_tablet_manager.cc:1151] Deleting tablet 5f5ebd2bf6614980ac0c18f5d9974fc0
1029 08:52:30.929078 (+ 76528us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.268000 (+338922us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.382274 (+114274us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.628675 (+246401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5179980}
I20251029 08:52:31.629211 1176 ts_tablet_manager.cc:1916] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.629266 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1037) took 5956 ms. Trace:
I20251029 08:52:31.629333 1974 rpcz_store.cc:276] 1029 08:52:25.672719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.672758 (+ 39us) service_pool.cc:225] Handling call
1029 08:52:31.629146 (+5956388us) ts_tablet_manager.cc:1151] Deleting tablet aafa846cb2144cec9ce4b28e61d91961
1029 08:52:31.629158 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.629250 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956278}
I20251029 08:52:31.629587 1974 tablet_replica.cc:333] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:31.629801 1974 raft_consensus.cc:2243] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.629976 1974 raft_consensus.cc:2272] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.630470 1974 ts_tablet_manager.cc:1916] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.632594 886 tablet_service.cc:1505] Processing CreateTablet for tablet 59348e20825e4be69b86738e6f6b79ba (DEFAULT_TABLE table=default.Table_6d00dde368eb4b14bb49537997d2a49f [id=c0528b1715cb4f1b89c04db3f8eba6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.633000 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59348e20825e4be69b86738e6f6b79ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.636309 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff85d521ec9f4f09b73ca0ea6541cd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.640247 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c601d8c3434b02ac480be7f59cc60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.644160 1178 tablet_bootstrap.cc:492] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.644308 1178 ts_tablet_manager.cc:1403] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.023s
I20251029 08:52:31.645445 1178 raft_consensus.cc:359] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.645628 1178 raft_consensus.cc:385] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.645668 1178 raft_consensus.cc:740] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.645857 1178 consensus_queue.cc:260] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.646423 1178 ts_tablet_manager.cc:1434] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:31.650017 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d010353b251140e99ce2df996f965991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.657027 1179 ts_tablet_manager.cc:1950] T dbf9e36c47be4509a998be8e7138475b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.658119 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1207) took 1572 ms. Trace:
I20251029 08:52:31.658489 2869 raft_consensus.cc:493] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.658640 2869 raft_consensus.cc:515] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.659374 2869 leader_election.cc:290] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.658309 1179 rpcz_store.cc:276] 1029 08:52:30.085591 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.131811 (+ 46220us) service_pool.cc:225] Handling call
1029 08:52:31.502160 (+1370349us) ts_tablet_manager.cc:1151] Deleting tablet dbf9e36c47be4509a998be8e7138475b
1029 08:52:31.502170 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.573827 (+ 71657us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.577016 (+ 3189us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.658093 (+ 81077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1370070}
I20251029 08:52:31.660261 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1209) took 1470 ms. Trace:
I20251029 08:52:31.660390 1179 rpcz_store.cc:276] 1029 08:52:30.190297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.190818 (+ 521us) service_pool.cc:225] Handling call
1029 08:52:31.660113 (+1469295us) ts_tablet_manager.cc:1151] Deleting tablet e27701fbf5b44e90ba33bed99fdc4d9b
1029 08:52:31.660126 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.660244 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469116}
I20251029 08:52:31.660836 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1210) took 1468 ms. Trace:
I20251029 08:52:31.660959 1179 rpcz_store.cc:276] 1029 08:52:30.192399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.192450 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:31.660704 (+1468254us) ts_tablet_manager.cc:1151] Deleting tablet 9e44e56554574d5abb67d08d88218152
1029 08:52:31.660714 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.660822 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468077}
I20251029 08:52:31.661306 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1208) took 1473 ms. Trace:
I20251029 08:52:31.661451 1179 rpcz_store.cc:276] 1029 08:52:30.188056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.197072 (+ 9016us) service_pool.cc:225] Handling call
1029 08:52:31.661221 (+1464149us) ts_tablet_manager.cc:1151] Deleting tablet 3ee83f5526f74ba2a344edd97db94050
1029 08:52:31.661229 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.661294 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463910}
I20251029 08:52:31.661995 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1211) took 1378 ms. Trace:
I20251029 08:52:31.662120 1179 rpcz_store.cc:276] 1029 08:52:30.283825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.283881 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:31.661907 (+1378026us) ts_tablet_manager.cc:1151] Deleting tablet 0adba75cc7d74b1497de9af7ad7ccdd0
1029 08:52:31.661915 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.661983 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377788}
I20251029 08:52:31.662408 1179 tablet_replica.cc:333] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.662627 1179 raft_consensus.cc:2243] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.662834 1179 raft_consensus.cc:2272] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.663429 1179 ts_tablet_manager.cc:1916] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.663918 1176 ts_tablet_manager.cc:1929] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.670336 1179 ts_tablet_manager.cc:1929] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.670575 1179 log.cc:1199] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9907d044daa145168417371c2bd66544
I20251029 08:52:31.670950 1179 ts_tablet_manager.cc:1950] T 9907d044daa145168417371c2bd66544 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.671824 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0d5ee0b03ce840c5ac78d83a1054511a (DEFAULT_TABLE table=default.Table_301169b3168b41b789340c0724a5afcd [id=dde2397e469d41819322a39aad089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.671962 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1219) took 1142 ms. Trace:
I20251029 08:52:31.672250 1179 rpcz_store.cc:276] 1029 08:52:30.529212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.529533 (+ 321us) service_pool.cc:225] Handling call
1029 08:52:31.662369 (+1132836us) ts_tablet_manager.cc:1151] Deleting tablet 9907d044daa145168417371c2bd66544
1029 08:52:31.662376 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.666873 (+ 4497us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.670220 (+ 3347us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.671942 (+ 1722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131678}
I20251029 08:52:31.672751 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1223) took 1007 ms. Trace:
I20251029 08:52:31.672911 1179 rpcz_store.cc:276] 1029 08:52:30.664946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.665003 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:31.672620 (+1007617us) ts_tablet_manager.cc:1151] Deleting tablet 7492195a41664ef69554d8a6cb148e21
1029 08:52:31.672630 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.672733 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007446}
I20251029 08:52:31.673314 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1222) took 1010 ms. Trace:
I20251029 08:52:31.673597 1179 rpcz_store.cc:276] 1029 08:52:30.662423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.666523 (+ 4100us) service_pool.cc:225] Handling call
1029 08:52:31.673213 (+1006690us) ts_tablet_manager.cc:1151] Deleting tablet d87191d05d4145a796a8823f723bc499
1029 08:52:31.673224 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.673300 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006518}
I20251029 08:52:31.673935 1179 tablet_replica.cc:333] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.674171 1179 raft_consensus.cc:2243] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.674392 1179 raft_consensus.cc:2272] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.675031 1179 ts_tablet_manager.cc:1916] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.676146 1176 log.cc:1199] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2fb9ab76dcfe486e89bbe9391cab56f9
I20251029 08:52:31.676541 1176 ts_tablet_manager.cc:1950] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:31.677472 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1178) took 1883 ms. Trace:
I20251029 08:52:31.677577 1176 rpcz_store.cc:276] 1029 08:52:29.793795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.793846 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:31.628139 (+1834293us) ts_tablet_manager.cc:1151] Deleting tablet 2fb9ab76dcfe486e89bbe9391cab56f9
1029 08:52:31.628151 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.660643 (+ 32492us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.663807 (+ 3164us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.677444 (+ 13637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815469}
I20251029 08:52:31.682298 1179 ts_tablet_manager.cc:1929] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.682494 1179 log.cc:1199] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/eeabf8f23751447aa9af02363c40361d
I20251029 08:52:31.682879 1179 ts_tablet_manager.cc:1950] T eeabf8f23751447aa9af02363c40361d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.683696 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1225) took 1015 ms. Trace:
I20251029 08:52:31.683856 1179 rpcz_store.cc:276] 1029 08:52:30.668244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.668307 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:31.673896 (+1005589us) ts_tablet_manager.cc:1151] Deleting tablet eeabf8f23751447aa9af02363c40361d
1029 08:52:31.673904 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.678888 (+ 4984us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.682194 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.683674 (+ 1480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005428}
I20251029 08:52:31.684280 1179 tablet_replica.cc:333] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.684487 1179 raft_consensus.cc:2243] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.684381 1176 tablet_replica.cc:333] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:31.684718 1179 raft_consensus.cc:2272] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.685272 1179 ts_tablet_manager.cc:1916] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.686079 1176 raft_consensus.cc:2243] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.687376 1176 raft_consensus.cc:2272] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.689289 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 5f5ebd2bf6614980ac0c18f5d9974fc0 (table default.Table_be549a784efc4332bf662c497d35ce49 [id=bb9fd353bed143dfa391ee55e8d09d01]) successfully deleted
W20251029 08:52:31.690659 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aafa846cb2144cec9ce4b28e61d91961 because the tablet was not found. No further retry: Not found: Tablet not found: aafa846cb2144cec9ce4b28e61d91961
I20251029 08:52:31.691542 2869 raft_consensus.cc:493] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.691690 2869 raft_consensus.cc:515] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.691864 1176 ts_tablet_manager.cc:1916] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.692633 2869 leader_election.cc:290] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.692975 1179 ts_tablet_manager.cc:1929] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.693127 1179 log.cc:1199] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/59ad04f57c29407b9d2420520c095a28
I20251029 08:52:31.693544 1179 ts_tablet_manager.cc:1950] T 59ad04f57c29407b9d2420520c095a28 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.694599 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1227) took 1024 ms. Trace:
I20251029 08:52:31.694753 1179 rpcz_store.cc:276] 1029 08:52:30.670422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.670677 (+ 255us) service_pool.cc:225] Handling call
1029 08:52:31.684228 (+1013551us) ts_tablet_manager.cc:1151] Deleting tablet 59ad04f57c29407b9d2420520c095a28
1029 08:52:31.684238 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.688380 (+ 4142us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.691397 (+ 3017us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.694575 (+ 3178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013378}
I20251029 08:52:31.695371 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1228) took 1024 ms. Trace:
I20251029 08:52:31.695501 1179 rpcz_store.cc:276] 1029 08:52:30.670960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.671138 (+ 178us) service_pool.cc:225] Handling call
1029 08:52:31.695222 (+1024084us) ts_tablet_manager.cc:1151] Deleting tablet 4dec9fdf1ae94ae687f30f89096ce89b
1029 08:52:31.695234 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.695352 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023916}
I20251029 08:52:31.695937 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1229) took 1022 ms. Trace:
I20251029 08:52:31.698477 1179 rpcz_store.cc:276] 1029 08:52:30.673927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.674038 (+ 111us) service_pool.cc:225] Handling call
1029 08:52:31.695806 (+1021768us) ts_tablet_manager.cc:1151] Deleting tablet 07bf5d693be240f18bacccc81a6ea5d4
1029 08:52:31.695815 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.695920 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021600}
I20251029 08:52:31.698870 1179 tablet_replica.cc:333] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.699103 1179 raft_consensus.cc:2243] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.699201 894 tablet_service.cc:1505] Processing CreateTablet for tablet 55710ce1ddd140d2a390cc5d89e8cfac (DEFAULT_TABLE table=default.Table_620d0574d4154375bbf3774c5ad37097 [id=b84b5834b26b4c3c8129cedb169c6b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.699318 1179 raft_consensus.cc:2272] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.700058 1179 ts_tablet_manager.cc:1916] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.701083 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "308da528313044929b66040d6659f774" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.701444 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485c9946f3f14a2aab11f4cdaa70a5b6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:31.702004 720 leader_election.cc:343] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:31.702257 720 leader_election.cc:343] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.704490 749 tablet_service.cc:1552] Processing DeleteTablet for tablet adb59d3ee338465a942e6b4423f0e071 with delete_type TABLET_DATA_DELETED (replaced by tablet afb32905315b4077926d028eaa404967) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.708215 1179 ts_tablet_manager.cc:1929] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.708478 1179 log.cc:1199] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0c2072c16fce43278f1e32b9672187e7
W20251029 08:52:31.708570 861 consensus_peers.cc:597] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ab56045bd814d59831a98618c759af4. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:31.708846 1179 ts_tablet_manager.cc:1950] T 0c2072c16fce43278f1e32b9672187e7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.709643 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37708 (request call id 1230) took 1022 ms. Trace:
I20251029 08:52:31.709830 1179 rpcz_store.cc:276] 1029 08:52:30.687074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.687299 (+ 225us) service_pool.cc:225] Handling call
1029 08:52:31.698825 (+1011526us) ts_tablet_manager.cc:1151] Deleting tablet 0c2072c16fce43278f1e32b9672187e7
1029 08:52:31.698835 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.704929 (+ 6094us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.708097 (+ 3168us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.709618 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011322}
I20251029 08:52:31.710853 1179 tablet_replica.cc:333] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.711099 1179 raft_consensus.cc:2243] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.711288 1179 raft_consensus.cc:2272] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.711930 1179 ts_tablet_manager.cc:1916] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.719044 1179 ts_tablet_manager.cc:1929] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.719270 1179 log.cc:1199] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/663d6587a5e14519afbb0f94d691d1b4
I20251029 08:52:31.719691 1179 ts_tablet_manager.cc:1950] T 663d6587a5e14519afbb0f94d691d1b4 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.721272 1179 tablet_replica.cc:333] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.721571 1179 raft_consensus.cc:2243] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:31.721956 1179 raft_consensus.cc:2272] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.722589 1179 ts_tablet_manager.cc:1916] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.723342 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet dbf9e36c47be4509a998be8e7138475b (table default.Table_47b07a98fcc34ec2a2784a43b313f904 [id=0265c5ac448a46a49a1bfcca4a305645]) successfully deleted
W20251029 08:52:31.723613 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e27701fbf5b44e90ba33bed99fdc4d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e27701fbf5b44e90ba33bed99fdc4d9b already in progress: creating tablet
W20251029 08:52:31.723810 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9e44e56554574d5abb67d08d88218152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e44e56554574d5abb67d08d88218152 already in progress: creating tablet
I20251029 08:52:31.724627 750 tablet_service.cc:1505] Processing CreateTablet for tablet 760fcb61d5274487b11a760573dfca6a (DEFAULT_TABLE table=default.Table_5e9da1c94f3e437d9b2e61d9a51a18a8 [id=485f1a12bf564f58918e15156e15d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.725594 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760fcb61d5274487b11a760573dfca6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.729411 2869 raft_consensus.cc:493] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.729576 2869 raft_consensus.cc:515] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.730240 2869 leader_election.cc:290] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.730530 1179 ts_tablet_manager.cc:1929] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:31.730696 1179 log.cc:1199] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/e1e2be77182e4259b76d20cae4e9e478
I20251029 08:52:31.731141 1179 ts_tablet_manager.cc:1950] T e1e2be77182e4259b76d20cae4e9e478 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.732434 1179 tablet_replica.cc:333] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:31.732777 1179 raft_consensus.cc:2243] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.732992 1179 raft_consensus.cc:2272] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.733614 1179 ts_tablet_manager.cc:1916] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:31.723991 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 3ee83f5526f74ba2a344edd97db94050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee83f5526f74ba2a344edd97db94050 already in progress: creating tablet
W20251029 08:52:31.743258 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0adba75cc7d74b1497de9af7ad7ccdd0 already in progress: creating tablet
I20251029 08:52:31.744210 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9907d044daa145168417371c2bd66544 (table default.Table_325e6358252a40b0bdd59c1a0664ed7e [id=3bb1d67e45ea4d7fbc5f901e60cf7083]) successfully deleted
I20251029 08:52:31.744608 889 tablet_service.cc:1505] Processing CreateTablet for tablet f7acde70ec7f4998ab63a37d025d35cf (DEFAULT_TABLE table=default.Table_cddcc5d90ac4457fbbe4489988f438f7 [id=63551108a9694c11afa79b5eac707956]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:31.744874 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7492195a41664ef69554d8a6cb148e21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7492195a41664ef69554d8a6cb148e21 already in progress: creating tablet
W20251029 08:52:31.745553 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d87191d05d4145a796a8823f723bc499 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d87191d05d4145a796a8823f723bc499 already in progress: creating tablet
I20251029 08:52:31.746058 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet eeabf8f23751447aa9af02363c40361d (table default.Table_5fefde56f4da40239f0ccb837befc3db [id=ea6733c09e8f48d2b64292168b11d69a]) successfully deleted
I20251029 08:52:31.746479 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 59ad04f57c29407b9d2420520c095a28 (table default.Table_dc4e52894b5141c0b0297a471a1c6eef [id=9ed3edb092184d5f9e99b08520772790]) successfully deleted
I20251029 08:52:31.746584 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "308da528313044929b66040d6659f774" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.746976 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 4dec9fdf1ae94ae687f30f89096ce89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dec9fdf1ae94ae687f30f89096ce89b already in progress: creating tablet
I20251029 08:52:31.747545 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485c9946f3f14a2aab11f4cdaa70a5b6" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.747968 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.752467 592 catalog_manager.cc:5649] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:31.753149 592 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:31.755841 1178 tablet_bootstrap.cc:492] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.757336 904 tablet_service.cc:1505] Processing CreateTablet for tablet 760fcb61d5274487b11a760573dfca6a (DEFAULT_TABLE table=default.Table_5e9da1c94f3e437d9b2e61d9a51a18a8 [id=485f1a12bf564f58918e15156e15d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.758559 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:31.759951 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:31.760686 720 leader_election.cc:343] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.761206 1183 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:31.761345 1183 ts_tablet_manager.cc:1403] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.348s user 0.011s sys 0.001s
I20251029 08:52:31.762104 1183 raft_consensus.cc:359] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.762279 1183 raft_consensus.cc:385] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.762334 1183 raft_consensus.cc:740] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.766835 1177 tablet_bootstrap.cc:492] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:31.767102 1177 ts_tablet_manager.cc:1403] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.155s user 0.002s sys 0.002s
I20251029 08:52:31.767906 1177 raft_consensus.cc:359] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.768225 1177 raft_consensus.cc:385] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.768431 1177 raft_consensus.cc:740] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.769037 1177 consensus_queue.cc:260] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.770692 1177 ts_tablet_manager.cc:1434] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 08:52:31.772670 1183 consensus_queue.cc:260] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:31.772403 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 07bf5d693be240f18bacccc81a6ea5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07bf5d693be240f18bacccc81a6ea5d4 already in progress: creating tablet
W20251029 08:52:31.771401 718 leader_election.cc:343] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.776185 718 leader_election.cc:304] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.776474 2869 raft_consensus.cc:2749] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.776731 718 leader_election.cc:343] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.777602 718 leader_election.cc:304] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.778026 2869 raft_consensus.cc:2749] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:31.778062 718 leader_election.cc:343] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.778158 718 leader_election.cc:304] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.778362 2869 raft_consensus.cc:2749] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.780230 1178 tablet_bootstrap.cc:654] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.780735 1176 ts_tablet_manager.cc:1929] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.781273 1176 log.cc:1199] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d26ff78aa5f94ae49a197daf2be00aa5
I20251029 08:52:31.782948 1176 ts_tablet_manager.cc:1950] T d26ff78aa5f94ae49a197daf2be00aa5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:31.788395 1183 ts_tablet_manager.cc:1434] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20251029 08:52:31.792295 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1179) took 1997 ms. Trace:
I20251029 08:52:31.792964 1183 tablet_bootstrap.cc:492] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:31.792922 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d306be1d51674c2abc506d248e05a9c7 (table default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]) successfully deleted
I20251029 08:52:31.793840 1176 rpcz_store.cc:276] 1029 08:52:29.794315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.794365 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:31.683970 (+1889605us) ts_tablet_manager.cc:1151] Deleting tablet d26ff78aa5f94ae49a197daf2be00aa5
1029 08:52:31.684178 (+ 208us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.720142 (+ 35964us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.780306 (+ 60164us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.791779 (+ 11473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871231}
I20251029 08:52:31.800235 1176 tablet_replica.cc:333] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:31.801806 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (table default.Table_86364f11715846199b42645e6deeb2fb [id=7e89e19d5a384f8195c22592fe17e43d]) successfully deleted
I20251029 08:52:31.804699 1176 raft_consensus.cc:2243] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:31.807333 1176 raft_consensus.cc:2272] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:31.809962 1179 ts_tablet_manager.cc:1929] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:31.813665 1179 log.cc:1199] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d306be1d51674c2abc506d248e05a9c7
I20251029 08:52:31.814172 1179 ts_tablet_manager.cc:1950] T d306be1d51674c2abc506d248e05a9c7 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:31.815824 1178 tablet_bootstrap.cc:492] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.816095 1178 ts_tablet_manager.cc:1403] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.000s
I20251029 08:52:31.817082 1178 raft_consensus.cc:359] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.817394 1178 raft_consensus.cc:385] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.817471 1176 ts_tablet_manager.cc:1916] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:31.817824 2869 raft_consensus.cc:493] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.818809 2869 raft_consensus.cc:515] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.819530 2869 leader_election.cc:290] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.817504 1178 raft_consensus.cc:740] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.820956 1178 consensus_queue.cc:260] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.823932 1183 tablet_bootstrap.cc:654] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.826126 2869 raft_consensus.cc:493] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.826273 2869 raft_consensus.cc:515] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.826910 2869 leader_election.cc:290] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.827666 1178 ts_tablet_manager.cc:1434] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 08:52:31.828074 1178 tablet_bootstrap.cc:492] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.829903 1178 tablet_bootstrap.cc:654] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.832358 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58932f38c0d64aec9a02238eaea56a25" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:31.843888 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d26ff78aa5f94ae49a197daf2be00aa5 (table default.Table_5018010ef9af4f16b6388715c7f9fd03 [id=edda8e15b59b472886bb1eba8fec8739]) successfully deleted
I20251029 08:52:31.848331 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58932f38c0d64aec9a02238eaea56a25" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.848852 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.849547 718 leader_election.cc:343] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:31.849797 718 leader_election.cc:343] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.855054 1178 tablet_bootstrap.cc:492] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:31.855237 1178 ts_tablet_manager.cc:1403] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20251029 08:52:31.855563 764 tablet_service.cc:1505] Processing CreateTablet for tablet b9508a4d2fad4b129f9a88bcce1695da (DEFAULT_TABLE table=default.Table_2cd17906960e40f59deaa3646d9f9359 [id=77283fe127ec47968fb2881d17cfdbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.855988 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508a4d2fad4b129f9a88bcce1695da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.856109 1178 raft_consensus.cc:359] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.856274 1178 raft_consensus.cc:385] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.856333 1178 raft_consensus.cc:740] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.856917 1178 consensus_queue.cc:260] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.857620 1178 ts_tablet_manager.cc:1434] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:31.857924 1178 tablet_bootstrap.cc:492] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:31.860993 750 tablet_service.cc:1505] Processing CreateTablet for tablet 036558cc29514284a91866c0997ba012 (DEFAULT_TABLE table=default.Table_ecbebd3f5d9b42218bb0dd513c106df2 [id=a307fb6f1d0141fa8ab9211651832bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.863363 745 tablet_service.cc:1505] Processing CreateTablet for tablet 9d8beb60b87e44b5a698f1750b0777da (DEFAULT_TABLE table=default.Table_306a00708d9448639fca68a831b3bef3 [id=3eeb8db034484fe784e905bb8d7e2b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.864205 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8beb60b87e44b5a698f1750b0777da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.866948 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 036558cc29514284a91866c0997ba012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.867589 1183 tablet_bootstrap.cc:492] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
W20251029 08:52:31.867722 595 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2afefb80dc14001811ddb811f21fd6d consensus state has no leader
I20251029 08:52:31.867733 1183 ts_tablet_manager.cc:1403] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20251029 08:52:31.869267 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 82589b43556c46559f0de4af7e8d9db0
I20251029 08:52:31.869238 1183 raft_consensus.cc:359] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.869426 1183 raft_consensus.cc:385] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.869480 1183 raft_consensus.cc:740] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.869686 1183 consensus_queue.cc:260] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.871201 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet ac09bb4502e440368633bbc90e9ad2cf
I20251029 08:52:31.871891 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb0e63a0c67b4931bc2282a6d83b4db1" } modify_external_catalogs: true
I20251029 08:52:31.872072 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bb0e63a0c67b4931bc2282a6d83b4db1" } modify_external_catalogs: true
I20251029 08:52:31.871646 1183 ts_tablet_manager.cc:1434] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 08:52:31.872730 1183 tablet_bootstrap.cc:492] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:31.873863 596 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7af99e95e334279ac160ca7e0b85e77 consensus state has no leader
I20251029 08:52:31.874519 593 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_84ecf561646e4d788d12eccac9cb8646" } new_table_name: "default.Table_39f51419346c4056b420a37f1e5d0a72" modify_external_catalogs: true
I20251029 08:52:31.874305 1178 tablet_bootstrap.cc:654] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.874660 745 tablet_service.cc:1505] Processing CreateTablet for tablet 522d66b408634596acb1b19321e3bfa6 (DEFAULT_TABLE table=default.Table_44fb5557e46d4baf91ba74fd423b6604 [id=12f8dd9314bf44fcadeb88d27aad5f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.875443 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 522d66b408634596acb1b19321e3bfa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.878412 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b3c1dbf9db424137b9c0cec53bb32311" } modify_external_catalogs: true
I20251029 08:52:31.878618 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_b3c1dbf9db424137b9c0cec53bb32311" } modify_external_catalogs: true
I20251029 08:52:31.881904 763 tablet_service.cc:1505] Processing CreateTablet for tablet a138c89afa3e4b8194e21b4fbdf6b08b (DEFAULT_TABLE table=default.Table_95d0e45bf0384a5c93ef2b9a12f8275b [id=9f66745012904044854fa5dfa239f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.882380 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a138c89afa3e4b8194e21b4fbdf6b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.899984 886 tablet_service.cc:1552] Processing DeleteTablet for tablet adb59d3ee338465a942e6b4423f0e071 with delete_type TABLET_DATA_DELETED (replaced by tablet afb32905315b4077926d028eaa404967) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:31.902957 1183 tablet_bootstrap.cc:654] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.905346 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760fcb61d5274487b11a760573dfca6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.905582 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55710ce1ddd140d2a390cc5d89e8cfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.906044 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263a2a981cb44c04bc881f8656338760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.910267 1974 ts_tablet_manager.cc:1929] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:31.910485 1974 log.cc:1199] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/43c3074caee74069a8a7f1b87721d726
I20251029 08:52:31.911310 1974 ts_tablet_manager.cc:1950] T 43c3074caee74069a8a7f1b87721d726 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:31.912201 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70732c0cfe6a4478a8fe329312781234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.912469 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ee0b03ce840c5ac78d83a1054511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.913637 755 tablet_service.cc:1505] Processing CreateTablet for tablet dc059173c2eb4742a5acf38a00bb7ecf (DEFAULT_TABLE table=default.Table_bce55107e4124f29a2e90ebc9781adae [id=a6591d8bc98c40838355bb24545eebba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.914036 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc059173c2eb4742a5acf38a00bb7ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.914324 2869 raft_consensus.cc:493] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.914458 2869 raft_consensus.cc:515] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.915083 2869 leader_election.cc:290] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.916217 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62597189ecb047788b83788b514e9099" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.916321 759 tablet_service.cc:1505] Processing CreateTablet for tablet b3946192856e47aa8db939b8409965b9 (DEFAULT_TABLE table=default.Table_f41fe46951a049778b48cb9eb97e3fc2 [id=627566ac73de4097a20303e12d9270cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.916682 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3946192856e47aa8db939b8409965b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:31.917014 718 leader_election.cc:343] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.917857 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7774c358f14e1687043ee3b8cd6ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.920342 752 tablet_service.cc:1505] Processing CreateTablet for tablet 73a7a8f00df14c2b9d89e2bc357ad512 (DEFAULT_TABLE table=default.Table_30db36162b644be4af80133b2d49d317 [id=4b55b99dcc4d4e26937186b59838c8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.920743 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a7a8f00df14c2b9d89e2bc357ad512. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.921062 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4644312dee44fd87047fb78e24d344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.921670 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a24dac06544e509eeb4a1c6072ba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.920359 2869 raft_consensus.cc:493] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.924211 2869 raft_consensus.cc:515] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.924894 2869 leader_election.cc:290] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.922125 894 tablet_service.cc:1505] Processing CreateTablet for tablet b9508a4d2fad4b129f9a88bcce1695da (DEFAULT_TABLE table=default.Table_2cd17906960e40f59deaa3646d9f9359 [id=77283fe127ec47968fb2881d17cfdbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.928236 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11d8baf25334120a4a0327df5dff2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.925412 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42de32d87ebb451090e0310e36e64277" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:31.928551 1177 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:31.929097 718 leader_election.cc:343] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.930044 1177 tablet_bootstrap.cc:654] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:31.930213 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2343d583cd8a4e04964316f256e22b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.934763 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54e0b4b332b43f7a0a4ccf01d7816d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.936338 904 tablet_service.cc:1505] Processing CreateTablet for tablet 9d8beb60b87e44b5a698f1750b0777da (DEFAULT_TABLE table=default.Table_306a00708d9448639fca68a831b3bef3 [id=3eeb8db034484fe784e905bb8d7e2b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.936643 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f138b5e5a34239843fff6b69fc43b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.937459 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508a4d2fad4b129f9a88bcce1695da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.940802 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35db4974b5b94e1ea0df16a086f90ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.944283 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fe71e31dbf42288f66f4160a13ed90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.944952 763 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec8fa568e884fe9aef62eb837c96258 (DEFAULT_TABLE table=default.Table_edb90c7ab4f746b6b8fd878de2a107a9 [id=3d19e9fe081340aeb1181ceb8f3f30e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.945364 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec8fa568e884fe9aef62eb837c96258. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.953996 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7acde70ec7f4998ab63a37d025d35cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.954819 752 tablet_service.cc:1505] Processing CreateTablet for tablet 669c4ee4efcd40e3bf8c52d55356fcbd (DEFAULT_TABLE table=default.Table_5cf067a75b3d44d5833cbac537f853eb [id=574e4c6d057d4e52a8b743e2bb48e949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.955281 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669c4ee4efcd40e3bf8c52d55356fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.956068 759 tablet_service.cc:1505] Processing CreateTablet for tablet faf6c2ac75e445e6bd94de27653f2860 (DEFAULT_TABLE table=default.Table_351d968afa8d4394aae3dfc23405ff0b [id=61edbd3f7e204c20a2d2a385fa1b4c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.956454 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf6c2ac75e445e6bd94de27653f2860. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.957350 755 tablet_service.cc:1505] Processing CreateTablet for tablet 663fbecd5cd945dd9e3f64932c2e8363 (DEFAULT_TABLE table=default.Table_05c08c8aca93406dadacfdc745eae1e0 [id=82e0c3d0fc8b48389e5f48ded293cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.957744 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663fbecd5cd945dd9e3f64932c2e8363. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.958773 890 tablet_service.cc:1505] Processing CreateTablet for tablet 522d66b408634596acb1b19321e3bfa6 (DEFAULT_TABLE table=default.Table_44fb5557e46d4baf91ba74fd423b6604 [id=12f8dd9314bf44fcadeb88d27aad5f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.960430 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:31.961082 720 leader_election.cc:343] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.961201 720 leader_election.cc:304] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:31.962765 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 522d66b408634596acb1b19321e3bfa6 (DEFAULT_TABLE table=default.Table_44fb5557e46d4baf91ba74fd423b6604 [id=12f8dd9314bf44fcadeb88d27aad5f5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.962733 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 9d8beb60b87e44b5a698f1750b0777da (DEFAULT_TABLE table=default.Table_306a00708d9448639fca68a831b3bef3 [id=3eeb8db034484fe784e905bb8d7e2b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.962714 1046 tablet_service.cc:1505] Processing CreateTablet for tablet b9508a4d2fad4b129f9a88bcce1695da (DEFAULT_TABLE table=default.Table_2cd17906960e40f59deaa3646d9f9359 [id=77283fe127ec47968fb2881d17cfdbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.962939 2869 raft_consensus.cc:2749] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:31.962937 1045 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec8fa568e884fe9aef62eb837c96258 (DEFAULT_TABLE table=default.Table_edb90c7ab4f746b6b8fd878de2a107a9 [id=3d19e9fe081340aeb1181ceb8f3f30e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.963116 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 522d66b408634596acb1b19321e3bfa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.963619 1183 tablet_bootstrap.cc:492] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:31.963735 1183 ts_tablet_manager.cc:1403] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.001s
I20251029 08:52:31.964510 1183 raft_consensus.cc:359] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.964690 1183 raft_consensus.cc:385] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:31.964741 1183 raft_consensus.cc:740] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:31.964972 1183 consensus_queue.cc:260] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.965581 1183 ts_tablet_manager.cc:1434] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:31.959573 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8beb60b87e44b5a698f1750b0777da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.966606 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf510abf5b6496c87e586592c32c3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.968328 1036 tablet_service.cc:1505] Processing CreateTablet for tablet a138c89afa3e4b8194e21b4fbdf6b08b (DEFAULT_TABLE table=default.Table_95d0e45bf0384a5c93ef2b9a12f8275b [id=9f66745012904044854fa5dfa239f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.970026 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9508a4d2fad4b129f9a88bcce1695da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.970408 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3631548b36644528bde2c3a2614c44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.970876 2869 raft_consensus.cc:493] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:31.971004 2869 raft_consensus.cc:515] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:31.971488 756 tablet_service.cc:1505] Processing CreateTablet for tablet 31c7b7fe77624dcf9b04a38948d3919c (DEFAULT_TABLE table=default.Table_17212839de0141b18ba288225b77ef1a [id=41e57d0e487841e199c2fa2ee2a46a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.971637 2869 leader_election.cc:290] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:31.971863 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7b7fe77624dcf9b04a38948d3919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.972167 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c4e208427b94bd79bcb5c1080647731" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:31.972672 718 leader_election.cc:343] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:31.978021 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec8fa568e884fe9aef62eb837c96258. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.978760 762 tablet_service.cc:1505] Processing CreateTablet for tablet 802a005d131f48b8b768f97e57038000 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.978870 758 tablet_service.cc:1505] Processing CreateTablet for tablet 6619c659ed8c43abb71c4222490ae0d9 (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.979094 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802a005d131f48b8b768f97e57038000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.979329 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619c659ed8c43abb71c4222490ae0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.979514 753 tablet_service.cc:1505] Processing CreateTablet for tablet 32d1699e88874c05b2ffadd2cfd4b252 (DEFAULT_TABLE table=default.Table_6a613283962a4289a60037f57d6c1e1a [id=6d8e26bcd4a64c9583d6257c8c62789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.979800 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d1699e88874c05b2ffadd2cfd4b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.983745 756 tablet_service.cc:1505] Processing CreateTablet for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b (DEFAULT_TABLE table=default.Table_03738991ceac4afc818190d455239870 [id=00f1697b9944498e9645a4aa2285a4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.912079 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1038) took 6239 ms. Trace:
I20251029 08:52:31.984134 1974 rpcz_store.cc:276] 1029 08:52:25.672974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.673011 (+ 37us) service_pool.cc:225] Handling call
1029 08:52:31.629537 (+5956526us) ts_tablet_manager.cc:1151] Deleting tablet 43c3074caee74069a8a7f1b87721d726
1029 08:52:31.629546 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.685017 (+ 55471us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.910163 (+225146us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:31.912060 (+ 1897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956412}
I20251029 08:52:31.984638 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 43c3074caee74069a8a7f1b87721d726 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]) successfully deleted
I20251029 08:52:31.993853 1183 tablet_bootstrap.cc:492] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:31.993980 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a138c89afa3e4b8194e21b4fbdf6b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:31.902483 886 tablet_service.cc:1505] Processing CreateTablet for tablet 036558cc29514284a91866c0997ba012 (DEFAULT_TABLE table=default.Table_ecbebd3f5d9b42218bb0dd513c106df2 [id=a307fb6f1d0141fa8ab9211651832bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:31.997911 1183 tablet_bootstrap.cc:654] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.000169 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8beb60b87e44b5a698f1750b0777da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.003983 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_bc2ad33f108841bf8d203c9182d05644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.005421 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc2ad33f108841bf8d203c9182d05644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.004227 590 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet a17142ffce3e4c359e715ef63747c40d
I20251029 08:52:32.007258 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_cfb3215efbfa410a8c53f7adfcbe6d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.007732 590 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb3215efbfa410a8c53f7adfcbe6d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.008332 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d010353b251140e99ce2df996f965991
I20251029 08:52:32.004858 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_082ce20f2d2a450f9216c2c34a03f696"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:32.008930 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_750150daf42a46f0a5f4f2a4d8925dd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.009186 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_082ce20f2d2a450f9216c2c34a03f696 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:32.009521 588 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_750150daf42a46f0a5f4f2a4d8925dd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:32.010490 593 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3915fd705caa420eb64879ea513b8cea consensus state has no leader
I20251029 08:52:32.010795 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_10da20471c1a49ec93301ed22cf79f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 08:52:32.011170 595 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb] alter complete (version 1)
I20251029 08:52:32.004287 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_6f7db122e41f478290349f3c2a281ac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.011718 592 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f7db122e41f478290349f3c2a281ac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.011627 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_e0e268e3dcf941c18ac578d3de6c40c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.012054 595 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e268e3dcf941c18ac578d3de6c40c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:32.011176 593 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10da20471c1a49ec93301ed22cf79f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.016410 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_982b0112dd78419e8ea54b9a114951b9" } new_table_name: "default.Table_808b5eec406b4b30b01c91078b48ff88" modify_external_catalogs: true
I20251029 08:52:32.029139 1177 tablet_bootstrap.cc:492] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:32.029433 1177 ts_tablet_manager.cc:1403] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.180s user 0.003s sys 0.002s
I20251029 08:52:32.030217 1177 raft_consensus.cc:359] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.030467 1177 raft_consensus.cc:385] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.030568 1177 raft_consensus.cc:740] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.030865 1177 consensus_queue.cc:260] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.031538 1177 ts_tablet_manager.cc:1434] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:32.036247 1183 tablet_bootstrap.cc:492] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.036442 1183 ts_tablet_manager.cc:1403] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20251029 08:52:32.037340 1183 raft_consensus.cc:359] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.037590 1183 raft_consensus.cc:385] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.037693 1183 raft_consensus.cc:740] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.037993 1183 consensus_queue.cc:260] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.038635 1183 ts_tablet_manager.cc:1434] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:32.038972 1183 tablet_bootstrap.cc:492] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.040614 1183 tablet_bootstrap.cc:654] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.045620 2896 consensus_queue.cc:1048] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:32.056267 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.059440 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 663fbecd5cd945dd9e3f64932c2e8363 (DEFAULT_TABLE table=default.Table_05c08c8aca93406dadacfdc745eae1e0 [id=82e0c3d0fc8b48389e5f48ded293cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.059624 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 32d1699e88874c05b2ffadd2cfd4b252 (DEFAULT_TABLE table=default.Table_6a613283962a4289a60037f57d6c1e1a [id=6d8e26bcd4a64c9583d6257c8c62789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.059850 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663fbecd5cd945dd9e3f64932c2e8363. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.059814 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b (DEFAULT_TABLE table=default.Table_03738991ceac4afc818190d455239870 [id=00f1697b9944498e9645a4aa2285a4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.060117 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.059440 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 31c7b7fe77624dcf9b04a38948d3919c (DEFAULT_TABLE table=default.Table_17212839de0141b18ba288225b77ef1a [id=41e57d0e487841e199c2fa2ee2a46a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.060278 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d1699e88874c05b2ffadd2cfd4b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.060220 1051 tablet_service.cc:1505] Processing CreateTablet for tablet 802a005d131f48b8b768f97e57038000 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.060497 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7b7fe77624dcf9b04a38948d3919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.064412 1051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802a005d131f48b8b768f97e57038000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.064453 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62597189ecb047788b83788b514e9099" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.064814 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42de32d87ebb451090e0310e36e64277" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.065057 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c4e208427b94bd79bcb5c1080647731" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.065173 1065 raft_consensus.cc:2468] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 0.
W20251029 08:52:32.065418 720 leader_election.cc:343] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.065532 720 leader_election.cc:304] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.066066 2869 raft_consensus.cc:2749] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.060122 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6619c659ed8c43abb71c4222490ae0d9 (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.066995 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619c659ed8c43abb71c4222490ae0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.068298 594 master_service.cc:954] ReplaceTablet: received request to replace tablet a08630cbc401467a9dde1ce1e198cef7 from {username='slave'} at 127.0.0.1:39506
W20251029 08:52:32.069392 720 leader_election.cc:343] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.069490 720 leader_election.cc:304] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.070039 2869 raft_consensus.cc:2749] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.070909 1046 tablet_service.cc:1505] Processing CreateTablet for tablet faf6c2ac75e445e6bd94de27653f2860 (DEFAULT_TABLE table=default.Table_351d968afa8d4394aae3dfc23405ff0b [id=61edbd3f7e204c20a2d2a385fa1b4c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.071229 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf6c2ac75e445e6bd94de27653f2860. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.072731 720 leader_election.cc:343] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.072851 720 leader_election.cc:304] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.073098 2920 raft_consensus.cc:2749] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.073272 720 leader_election.cc:304] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.073838 2920 raft_consensus.cc:2804] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:32.073978 2920 raft_consensus.cc:493] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:32.074159 2920 raft_consensus.cc:3060] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.077069 2920 raft_consensus.cc:515] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.078038 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c4e208427b94bd79bcb5c1080647731" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee"
I20251029 08:52:32.078200 1065 raft_consensus.cc:3060] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.078677 2920 leader_election.cc:290] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.080158 1065 raft_consensus.cc:2468] T 6c4e208427b94bd79bcb5c1080647731 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8d80098fcb40ada903c3b2bacc2ea0 in term 1.
I20251029 08:52:32.080319 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 669c4ee4efcd40e3bf8c52d55356fcbd (DEFAULT_TABLE table=default.Table_5cf067a75b3d44d5833cbac537f853eb [id=574e4c6d057d4e52a8b743e2bb48e949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.080569 720 leader_election.cc:304] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:32.080763 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669c4ee4efcd40e3bf8c52d55356fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.080897 2920 raft_consensus.cc:2804] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:32.081091 2920 raft_consensus.cc:697] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Running, Role: LEADER
I20251029 08:52:32.081526 2920 consensus_queue.cc:237] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.085265 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 73a7a8f00df14c2b9d89e2bc357ad512 (DEFAULT_TABLE table=default.Table_30db36162b644be4af80133b2d49d317 [id=4b55b99dcc4d4e26937186b59838c8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.085655 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a7a8f00df14c2b9d89e2bc357ad512. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.087975 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac (DEFAULT_TABLE table=default.Table_71e81aba893d4388889abc1b3d7afebb [id=756090766f4d44eab7de18355541953c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.088389 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.088567 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c2f28c824c23440fb7489d34a117db4e (DEFAULT_TABLE table=default.Table_e0618293cfb14d9ab93a0f5f6fb1dd66 [id=885797dd36d247228908c75f5b8140ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.088907 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f28c824c23440fb7489d34a117db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.088917 1036 tablet_service.cc:1505] Processing CreateTablet for tablet b3946192856e47aa8db939b8409965b9 (DEFAULT_TABLE table=default.Table_f41fe46951a049778b48cb9eb97e3fc2 [id=627566ac73de4097a20303e12d9270cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.092362 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 308b058957c043d08b22a6dd1111c7ff (DEFAULT_TABLE table=default.Table_18732ceb5df84aa88ea943c4d35e6084 [id=dc3fbd651109405fbbb8d9e1a3269087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.092645 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 66752ba798f54b9790d0750219e4c4e9 (DEFAULT_TABLE table=default.Table_39ab2d051bdd4c3a8bbf20b2178eba44 [id=0565582732a14bc6b9f491683d7b54b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.093509 762 tablet_service.cc:1505] Processing CreateTablet for tablet c2f28c824c23440fb7489d34a117db4e (DEFAULT_TABLE table=default.Table_e0618293cfb14d9ab93a0f5f6fb1dd66 [id=885797dd36d247228908c75f5b8140ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.095547 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f28c824c23440fb7489d34a117db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.096894 588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bbe82478d4d64216b1b2fbc3e9c3aade" } modify_external_catalogs: true
I20251029 08:52:32.096860 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5dc644387108496995ffad8f788f3c24" } new_table_name: "default.Table_dfe031dd8c3e4be5b07e985ec97a8aae" modify_external_catalogs: true
I20251029 08:52:32.097283 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_ec5782dfc71a40e5870d6fc9a8e51780" } new_table_name: "default.Table_7f0e7183622649208e83f7e5e30cde04" modify_external_catalogs: true
I20251029 08:52:32.097324 588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_bbe82478d4d64216b1b2fbc3e9c3aade" } modify_external_catalogs: true
I20251029 08:52:32.098600 2869 raft_consensus.cc:493] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.098742 2869 raft_consensus.cc:515] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.099428 2869 leader_election.cc:290] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.100003 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:32.100688 720 leader_election.cc:343] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.092486 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3946192856e47aa8db939b8409965b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.101264 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b058957c043d08b22a6dd1111c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.101533 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66752ba798f54b9790d0750219e4c4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.107149 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5f0752745f5b45c7a3e2be18e3fac5a9" } modify_external_catalogs: true
I20251029 08:52:32.107355 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_5f0752745f5b45c7a3e2be18e3fac5a9" } modify_external_catalogs: true
I20251029 08:52:32.113463 1183 tablet_bootstrap.cc:492] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.118938 1183 ts_tablet_manager.cc:1403] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20251029 08:52:32.119985 1183 raft_consensus.cc:359] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.120244 1183 raft_consensus.cc:385] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.120321 1183 raft_consensus.cc:740] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.120594 1183 consensus_queue.cc:260] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.121150 1183 ts_tablet_manager.cc:1434] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:32.122797 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 3990e2983ffe4f1ca26f68875416a72a (DEFAULT_TABLE table=default.Table_1c0fa4716ab14f70b36d4f0ffe00fb93 [id=bb78195d9cd04c43a6f91288912631d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.123241 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990e2983ffe4f1ca26f68875416a72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.123618 1044 tablet_service.cc:1505] Processing CreateTablet for tablet 8837d051b71d4dd08557bee10d119ae9 (DEFAULT_TABLE table=default.Table_b0e8b56db4984d038e0b83a4948bd8b5 [id=d5758c6e20a74ccf9a9ebe2774c0a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.123992 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8837d051b71d4dd08557bee10d119ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.126660 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca (DEFAULT_TABLE table=default.Table_a6e6d2c56981480d877ea3030fcde485 [id=439ec3c69fdc42369a31ebc725a3053c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.127095 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.138437 1041 tablet_service.cc:1505] Processing CreateTablet for tablet ca2cab01089e43b5b8fc20d7cf7a475d (DEFAULT_TABLE table=default.Table_3b11dfccad424815b9e7803c8c2dd90b [id=4284c3adf4b540d3b87ba46eeb0adf61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.138463 753 tablet_service.cc:1505] Processing CreateTablet for tablet 8837d051b71d4dd08557bee10d119ae9 (DEFAULT_TABLE table=default.Table_b0e8b56db4984d038e0b83a4948bd8b5 [id=d5758c6e20a74ccf9a9ebe2774c0a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.138834 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2cab01089e43b5b8fc20d7cf7a475d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.138870 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8837d051b71d4dd08557bee10d119ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.146895 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 17f8f822a547460990eaa2544f0a3aff (DEFAULT_TABLE table=default.Table_e3c95dd5d9224e2d8bddacd4dc0be319 [id=0cf6846fdb4947e2acccb6aa7aa3ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.147105 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a (DEFAULT_TABLE table=default.Table_7da2a2ba63ee4190bbe7e36d1cac99bf [id=d0ed5cacdac54c3bb391a187751c4c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.147265 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f8f822a547460990eaa2544f0a3aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.147442 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.147691 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 82589b43556c46559f0de4af7e8d9db0 (DEFAULT_TABLE table=default.Table_43aa92e7fba74d91ad0b832e5f5d6592 [id=f8ae6d6bbf1944e5bd793db22dcdfd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.147974 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82589b43556c46559f0de4af7e8d9db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.148343 758 tablet_service.cc:1505] Processing CreateTablet for tablet 17f8f822a547460990eaa2544f0a3aff (DEFAULT_TABLE table=default.Table_e3c95dd5d9224e2d8bddacd4dc0be319 [id=0cf6846fdb4947e2acccb6aa7aa3ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.148663 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f8f822a547460990eaa2544f0a3aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.150689 1050 tablet_service.cc:1505] Processing CreateTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 (DEFAULT_TABLE table=default.Table_e229e75b9314413f8e7b3696ee9e6f55 [id=32ab0b598e01451c8c64bd8c884f5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.150987 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7818ad8e5ec4fca8e3b25019f545dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.151484 754 tablet_service.cc:1505] Processing CreateTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 (DEFAULT_TABLE table=default.Table_e229e75b9314413f8e7b3696ee9e6f55 [id=32ab0b598e01451c8c64bd8c884f5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.151832 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7818ad8e5ec4fca8e3b25019f545dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.121482 1183 tablet_bootstrap.cc:492] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.153556 1183 tablet_bootstrap.cc:654] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.154294 924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.156234 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c4e208427b94bd79bcb5c1080647731" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:32.157063 923 raft_consensus.cc:3060] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.162465 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 036558cc29514284a91866c0997ba012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.162942 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0c2072c16fce43278f1e32b9672187e7 (table default.Table_bceccce7081b43f6ad2849ed269dfe73 [id=468fee62b8ce4f0dad6500dcfe3c70c6]) successfully deleted
W20251029 08:52:32.163170 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc6c8f9ee9840c08c9436ab516a62d7 already in progress: creating tablet
W20251029 08:52:32.163326 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet e322478c9ba64c2a95a2c299123926dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e322478c9ba64c2a95a2c299123926dd already in progress: creating tablet
W20251029 08:52:32.163450 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 0b0f1cdfb1804d1c8d6220907795fc80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b0f1cdfb1804d1c8d6220907795fc80 already in progress: creating tablet
W20251029 08:52:32.163589 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 5ad98112c0ea4f59980e72165cef8a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ad98112c0ea4f59980e72165cef8a1f already in progress: creating tablet
I20251029 08:52:32.163707 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 663d6587a5e14519afbb0f94d691d1b4 (table default.Table_ab4e50ed7bad43b4adab2ec2cf76a327 [id=cf151f9ad5974db78cd578b1edfd034b]) successfully deleted
W20251029 08:52:32.163841 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8970ca1f227849a1a99ac5d33e450e14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8970ca1f227849a1a99ac5d33e450e14 already in progress: creating tablet
I20251029 08:52:32.166632 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 522d66b408634596acb1b19321e3bfa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.167354 761 tablet_service.cc:1505] Processing CreateTablet for tablet 82589b43556c46559f0de4af7e8d9db0 (DEFAULT_TABLE table=default.Table_43aa92e7fba74d91ad0b832e5f5d6592 [id=f8ae6d6bbf1944e5bd793db22dcdfd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.167750 746 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd6c279f9f4d7282d66e2e4101b668 (DEFAULT_TABLE table=default.Table_e52a0b50fc624679bc5d91a8cf424a20 [id=547c00ece3cf465a87d1e275010e78de]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.167914 718 leader_election.cc:343] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.168053 718 leader_election.cc:304] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.168220 760 tablet_service.cc:1505] Processing CreateTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 (DEFAULT_TABLE table=default.Table_bbe82478d4d64216b1b2fbc3e9c3aade [id=9d70d363be5540ac9c07d20efac755c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.168330 2869 raft_consensus.cc:2749] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:32.168422 718 leader_election.cc:343] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.168720 751 tablet_service.cc:1505] Processing CreateTablet for tablet e6403e337ac3477082ba730c5312e591 (DEFAULT_TABLE table=default.Table_532d58c219f94321bf5460c99c389e1e [id=bc4d0acb083046c6b72715c48235a72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.168982 757 tablet_service.cc:1505] Processing CreateTablet for tablet c21cfbfbde5541a8976aa9a9b50e8867 (DEFAULT_TABLE table=default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.167959 749 tablet_service.cc:1505] Processing CreateTablet for tablet 32929763e9b74f3ebb689e6ab5828443 (DEFAULT_TABLE table=default.Table_35877e0584a84248a87841ffcb0cbf94 [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.168220 748 tablet_service.cc:1505] Processing CreateTablet for tablet 19414da9e1d34c38ba28a1fc86c43c0d (DEFAULT_TABLE table=default.Table_ec5782dfc71a40e5870d6fc9a8e51780 [id=1ba323d0b8eb46a082394b69e7c3efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.167745 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82589b43556c46559f0de4af7e8d9db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.169467 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd6c279f9f4d7282d66e2e4101b668. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.169739 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19414da9e1d34c38ba28a1fc86c43c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.172224 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1e2c24e30048abb3ae9fb92171e049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.172250 2896 raft_consensus.cc:493] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.172387 2896 raft_consensus.cc:515] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.172564 898 tablet_service.cc:1505] Processing CreateTablet for tablet a138c89afa3e4b8194e21b4fbdf6b08b (DEFAULT_TABLE table=default.Table_95d0e45bf0384a5c93ef2b9a12f8275b [id=9f66745012904044854fa5dfa239f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.172902 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a138c89afa3e4b8194e21b4fbdf6b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.173027 2896 leader_election.cc:290] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.173219 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32929763e9b74f3ebb689e6ab5828443. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.176147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6403e337ac3477082ba730c5312e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.176337 747 tablet_service.cc:1505] Processing CreateTablet for tablet 33e4694f197b46a1bc2694bf8d91128c (DEFAULT_TABLE table=default.Table_f0938163b61540419aa4d8be600dcc1e [id=1d16258b1e974514be7a21a5782dcbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.177198 900 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec8fa568e884fe9aef62eb837c96258 (DEFAULT_TABLE table=default.Table_edb90c7ab4f746b6b8fd878de2a107a9 [id=3d19e9fe081340aeb1181ceb8f3f30e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.177526 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec8fa568e884fe9aef62eb837c96258. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.180428 897 tablet_service.cc:1505] Processing CreateTablet for tablet dc059173c2eb4742a5acf38a00bb7ecf (DEFAULT_TABLE table=default.Table_bce55107e4124f29a2e90ebc9781adae [id=a6591d8bc98c40838355bb24545eebba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.180763 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc059173c2eb4742a5acf38a00bb7ecf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.182240 582 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f2afefb80dc14001811ddb811f21fd6d (table default.Table_a7d94cf13cd648eaaddb49ebbbd78dc6 [id=7c8eae3dfd3e42a8888990aeadc118a9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2afefb80dc14001811ddb811f21fd6d consensus state has no leader
I20251029 08:52:32.183429 2869 raft_consensus.cc:493] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.183502 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85df43bf781c421688f163331de0eb2b" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.183555 2869 raft_consensus.cc:515] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.184460 2869 leader_election.cc:290] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.184403 902 tablet_service.cc:1505] Processing CreateTablet for tablet b3946192856e47aa8db939b8409965b9 (DEFAULT_TABLE table=default.Table_f41fe46951a049778b48cb9eb97e3fc2 [id=627566ac73de4097a20303e12d9270cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.184758 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3946192856e47aa8db939b8409965b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.184715 2896 raft_consensus.cc:493] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.184856 2896 raft_consensus.cc:515] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.185068 893 tablet_service.cc:1505] Processing CreateTablet for tablet faf6c2ac75e445e6bd94de27653f2860 (DEFAULT_TABLE table=default.Table_351d968afa8d4394aae3dfc23405ff0b [id=61edbd3f7e204c20a2d2a385fa1b4c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.185377 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf6c2ac75e445e6bd94de27653f2860. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.185452 2896 leader_election.cc:290] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.186874 1048 tablet_service.cc:1505] Processing CreateTablet for tablet 19414da9e1d34c38ba28a1fc86c43c0d (DEFAULT_TABLE table=default.Table_ec5782dfc71a40e5870d6fc9a8e51780 [id=1ba323d0b8eb46a082394b69e7c3efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.187245 1048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19414da9e1d34c38ba28a1fc86c43c0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.188288 974 tablet.cc:2378] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:32.184765 899 tablet_service.cc:1505] Processing CreateTablet for tablet 669c4ee4efcd40e3bf8c52d55356fcbd (DEFAULT_TABLE table=default.Table_5cf067a75b3d44d5833cbac537f853eb [id=574e4c6d057d4e52a8b743e2bb48e949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.189317 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669c4ee4efcd40e3bf8c52d55356fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.190675 1177 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:32.191874 1032 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd6c279f9f4d7282d66e2e4101b668 (DEFAULT_TABLE table=default.Table_e52a0b50fc624679bc5d91a8cf424a20 [id=547c00ece3cf465a87d1e275010e78de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.192193 1177 tablet_bootstrap.cc:654] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.192689 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd6c279f9f4d7282d66e2e4101b668. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.192816 2554 raft_consensus.cc:493] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.192955 2554 raft_consensus.cc:515] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.193046 1034 tablet_service.cc:1505] Processing CreateTablet for tablet b3de9bd5a00e42c28c8eadc36795b286 (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.193370 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9bd5a00e42c28c8eadc36795b286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.184412 904 tablet_service.cc:1505] Processing CreateTablet for tablet 73a7a8f00df14c2b9d89e2bc357ad512 (DEFAULT_TABLE table=default.Table_30db36162b644be4af80133b2d49d317 [id=4b55b99dcc4d4e26937186b59838c8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.193614 2554 leader_election.cc:290] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.193666 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a7a8f00df14c2b9d89e2bc357ad512. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.200644 861 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2fb380018f46465e82ac6223ecf8dd0d
W20251029 08:52:32.200883 861 leader_election.cc:343] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 2fb380018f46465e82ac6223ecf8dd0d
I20251029 08:52:32.200963 861 leader_election.cc:304] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:32.201077 1034 tablet_service.cc:1505] Processing CreateTablet for tablet ebc20228df5849219224d59c1370237b (DEFAULT_TABLE table=default.Table_cbee24e26857417db3a2cb72aca81cb5 [id=502c1023eff24760ac2629849642b5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.201555 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc20228df5849219224d59c1370237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.201623 2554 raft_consensus.cc:2749] T 2fb380018f46465e82ac6223ecf8dd0d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.199996 748 tablet_service.cc:1505] Processing CreateTablet for tablet b3de9bd5a00e42c28c8eadc36795b286 (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.206161 891 tablet_service.cc:1505] Processing CreateTablet for tablet 663fbecd5cd945dd9e3f64932c2e8363 (DEFAULT_TABLE table=default.Table_05c08c8aca93406dadacfdc745eae1e0 [id=82e0c3d0fc8b48389e5f48ded293cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.206341 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_2c16fc98c6a94cf7a42c8cb0413418d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.206804 998 leader_election.cc:343] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 85df43bf781c421688f163331de0eb2b
W20251029 08:52:32.206954 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c16fc98c6a94cf7a42c8cb0413418d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.206336 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4c4050fe044d04b76768d39ee7db7e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.208397 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663fbecd5cd945dd9e3f64932c2e8363. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.208395 2869 raft_consensus.cc:493] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.208539 2869 raft_consensus.cc:515] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.209165 2869 leader_election.cc:290] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.211845 1040 tablet_service.cc:1505] Processing CreateTablet for tablet c21cfbfbde5541a8976aa9a9b50e8867 (DEFAULT_TABLE table=default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.212298 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21cfbfbde5541a8976aa9a9b50e8867. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.214881 1043 tablet_service.cc:1505] Processing CreateTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 (DEFAULT_TABLE table=default.Table_bbe82478d4d64216b1b2fbc3e9c3aade [id=9d70d363be5540ac9c07d20efac755c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.215286 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1e2c24e30048abb3ae9fb92171e049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.218884 886 tablet_service.cc:1505] Processing CreateTablet for tablet 32d1699e88874c05b2ffadd2cfd4b252 (DEFAULT_TABLE table=default.Table_6a613283962a4289a60037f57d6c1e1a [id=6d8e26bcd4a64c9583d6257c8c62789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.219240 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d1699e88874c05b2ffadd2cfd4b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.219404 1050 tablet_service.cc:1505] Processing CreateTablet for tablet e6403e337ac3477082ba730c5312e591 (DEFAULT_TABLE table=default.Table_532d58c219f94321bf5460c99c389e1e [id=bc4d0acb083046c6b72715c48235a72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.219827 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6403e337ac3477082ba730c5312e591. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.221122 718 leader_election.cc:343] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.221298 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34729c3f97a44713a352bde297a80027" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:32.221576 998 leader_election.cc:343] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 2fb9ab76dcfe486e89bbe9391cab56f9
I20251029 08:52:32.223865 890 tablet_service.cc:1505] Processing CreateTablet for tablet 6619c659ed8c43abb71c4222490ae0d9 (DEFAULT_TABLE table=default.Table_f2f8ee81fa654340ba4535f2a11a0fb8 [id=89be4fee52d64aa7ad23a939f6f533a5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.224787 996 consensus_peers.cc:597] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:32.224794 718 leader_election.cc:343] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.224851 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619c659ed8c43abb71c4222490ae0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.224962 1176 ts_tablet_manager.cc:1929] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:32.225127 1176 log.cc:1199] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/4200db822deb4ace936f561c230ac1fb
W20251029 08:52:32.225174 996 leader_election.cc:343] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.225368 996 leader_election.cc:304] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.225675 2359 raft_consensus.cc:2749] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.225801 1176 ts_tablet_manager.cc:1950] T 4200db822deb4ace936f561c230ac1fb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:32.226306 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fb9ab76dcfe486e89bbe9391cab56f9" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.226814 1067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34729c3f97a44713a352bde297a80027" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.226887 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4c4050fe044d04b76768d39ee7db7e" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:32.227378 720 leader_election.cc:343] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.227494 720 leader_election.cc:304] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.227975 2869 raft_consensus.cc:2749] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.228610 905 tablet_service.cc:1505] Processing CreateTablet for tablet 549826233f6d49fd9bbddb70b6ade0d3 (DEFAULT_TABLE table=default.Table_f3d319733949430482ae8e14a2129b7b [id=f966541bd5b040719ae5c5218d46bb4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.229187 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 33e4694f197b46a1bc2694bf8d91128c (DEFAULT_TABLE table=default.Table_f0938163b61540419aa4d8be600dcc1e [id=1d16258b1e974514be7a21a5782dcbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.229389 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549826233f6d49fd9bbddb70b6ade0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.229625 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e4694f197b46a1bc2694bf8d91128c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.232293 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 32929763e9b74f3ebb689e6ab5828443 (DEFAULT_TABLE table=default.Table_35877e0584a84248a87841ffcb0cbf94 [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.232717 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32929763e9b74f3ebb689e6ab5828443. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.232913 1046 tablet_service.cc:1505] Processing CreateTablet for tablet 549826233f6d49fd9bbddb70b6ade0d3 (DEFAULT_TABLE table=default.Table_f3d319733949430482ae8e14a2129b7b [id=f966541bd5b040719ae5c5218d46bb4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.232919 902 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac (DEFAULT_TABLE table=default.Table_71e81aba893d4388889abc1b3d7afebb [id=756090766f4d44eab7de18355541953c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.233762 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.234230 755 tablet_service.cc:1505] Processing CreateTablet for tablet 66752ba798f54b9790d0750219e4c4e9 (DEFAULT_TABLE table=default.Table_39ab2d051bdd4c3a8bbf20b2178eba44 [id=0565582732a14bc6b9f491683d7b54b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.235108 900 tablet_service.cc:1505] Processing CreateTablet for tablet 308b058957c043d08b22a6dd1111c7ff (DEFAULT_TABLE table=default.Table_18732ceb5df84aa88ea943c4d35e6084 [id=dc3fbd651109405fbbb8d9e1a3269087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.235790 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b058957c043d08b22a6dd1111c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.236382 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1043) took 6477 ms. Trace:
I20251029 08:52:32.236507 1974 rpcz_store.cc:276] 1029 08:52:25.759295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.759348 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:32.076495 (+6317147us) ts_tablet_manager.cc:1151] Deleting tablet 0c2072c16fce43278f1e32b9672187e7
1029 08:52:32.236246 (+159751us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.236362 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6316977}
I20251029 08:52:32.237102 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1044) took 6477 ms. Trace:
I20251029 08:52:32.237026 763 tablet_service.cc:1505] Processing CreateTablet for tablet 549826233f6d49fd9bbddb70b6ade0d3 (DEFAULT_TABLE table=default.Table_f3d319733949430482ae8e14a2129b7b [id=f966541bd5b040719ae5c5218d46bb4b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.237114 720 leader_election.cc:343] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.237219 1974 rpcz_store.cc:276] 1029 08:52:25.759593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.759635 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:32.236957 (+6477322us) ts_tablet_manager.cc:1151] Deleting tablet 41622daeb1e0419fa88cc26c0405c2db
1029 08:52:32.236965 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.237087 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6477190}
I20251029 08:52:32.237336 720 leader_election.cc:304] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:32.237434 996 leader_election.cc:343] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.237529 996 leader_election.cc:304] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.237740 2359 raft_consensus.cc:2749] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.237843 2869 raft_consensus.cc:2749] T 9d4c4050fe044d04b76768d39ee7db7e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.237627 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1045) took 6477 ms. Trace:
I20251029 08:52:32.238186 1974 rpcz_store.cc:276] 1029 08:52:25.759985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.760047 (+ 62us) service_pool.cc:225] Handling call
1029 08:52:32.237532 (+6477485us) ts_tablet_manager.cc:1151] Deleting tablet 5498826a9afd4fd5a6b88b0a465fff40
1029 08:52:32.237540 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.237613 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6477353}
W20251029 08:52:32.238402 996 consensus_peers.cc:597] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:32.238407 759 tablet_service.cc:1505] Processing CreateTablet for tablet 3990e2983ffe4f1ca26f68875416a72a (DEFAULT_TABLE table=default.Table_1c0fa4716ab14f70b36d4f0ffe00fb93 [id=bb78195d9cd04c43a6f91288912631d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.238670 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1048) took 6424 ms. Trace:
I20251029 08:52:32.238756 1974 rpcz_store.cc:276] 1029 08:52:25.813919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.813983 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:32.238570 (+6424587us) ts_tablet_manager.cc:1151] Deleting tablet 58b02839973e4f018a1a6dfccf009b25
1029 08:52:32.238579 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.238656 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6424433}
W20251029 08:52:32.239126 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0c2072c16fce43278f1e32b9672187e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c2072c16fce43278f1e32b9672187e7 already in progress: creating tablet
W20251029 08:52:32.239360 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 41622daeb1e0419fa88cc26c0405c2db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41622daeb1e0419fa88cc26c0405c2db already in progress: creating tablet
W20251029 08:52:32.239660 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 58b02839973e4f018a1a6dfccf009b25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58b02839973e4f018a1a6dfccf009b25 already in progress: creating tablet
I20251029 08:52:32.240116 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1047) took 6426 ms. Trace:
I20251029 08:52:32.208007 899 tablet_service.cc:1505] Processing CreateTablet for tablet 31c7b7fe77624dcf9b04a38948d3919c (DEFAULT_TABLE table=default.Table_17212839de0141b18ba288225b77ef1a [id=41e57d0e487841e199c2fa2ee2a46a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.240541 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7b7fe77624dcf9b04a38948d3919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.241513 904 tablet_service.cc:1505] Processing CreateTablet for tablet c2f28c824c23440fb7489d34a117db4e (DEFAULT_TABLE table=default.Table_e0618293cfb14d9ab93a0f5f6fb1dd66 [id=885797dd36d247228908c75f5b8140ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.241889 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f28c824c23440fb7489d34a117db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.244299 897 tablet_service.cc:1505] Processing CreateTablet for tablet 802a005d131f48b8b768f97e57038000 (DEFAULT_TABLE table=default.Table_65e541ea2b244bf096d9362726620996 [id=a887add824284e16a3c7f995a8f088f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.244764 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802a005d131f48b8b768f97e57038000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.245514 893 tablet_service.cc:1505] Processing CreateTablet for tablet 66752ba798f54b9790d0750219e4c4e9 (DEFAULT_TABLE table=default.Table_39ab2d051bdd4c3a8bbf20b2178eba44 [id=0565582732a14bc6b9f491683d7b54b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.245878 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66752ba798f54b9790d0750219e4c4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.246634 903 tablet_service.cc:1505] Processing CreateTablet for tablet 3990e2983ffe4f1ca26f68875416a72a (DEFAULT_TABLE table=default.Table_1c0fa4716ab14f70b36d4f0ffe00fb93 [id=bb78195d9cd04c43a6f91288912631d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.236336 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac (DEFAULT_TABLE table=default.Table_71e81aba893d4388889abc1b3d7afebb [id=756090766f4d44eab7de18355541953c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.240061 898 tablet_service.cc:1505] Processing CreateTablet for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b (DEFAULT_TABLE table=default.Table_03738991ceac4afc818190d455239870 [id=00f1697b9944498e9645a4aa2285a4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.248230 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ca7a3f204f4588cb51b9fc68ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.248627 750 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a (DEFAULT_TABLE table=default.Table_7da2a2ba63ee4190bbe7e36d1cac99bf [id=d0ed5cacdac54c3bb391a187751c4c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.249198 1045 tablet_service.cc:1505] Processing CreateTablet for tablet dc059173c2eb4742a5acf38a00bb7ecf (DEFAULT_TABLE table=default.Table_bce55107e4124f29a2e90ebc9781adae [id=a6591d8bc98c40838355bb24545eebba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.249627 1045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc059173c2eb4742a5acf38a00bb7ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.251544 1046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549826233f6d49fd9bbddb70b6ade0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.200662 758 tablet_service.cc:1505] Processing CreateTablet for tablet ebc20228df5849219224d59c1370237b (DEFAULT_TABLE table=default.Table_cbee24e26857417db3a2cb72aca81cb5 [id=502c1023eff24760ac2629849642b5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.240229 1974 rpcz_store.cc:276] 1029 08:52:25.813689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.814833 (+ 1144us) service_pool.cc:225] Handling call
1029 08:52:32.239992 (+6425159us) ts_tablet_manager.cc:1151] Deleting tablet 52b697b962f849eb88741c2e2ba4643d
1029 08:52:32.240001 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.240102 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6425005}
I20251029 08:52:32.258255 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1046) took 6456 ms. Trace:
W20251029 08:52:32.258297 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 52b697b962f849eb88741c2e2ba4643d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52b697b962f849eb88741c2e2ba4643d already in progress: creating tablet
I20251029 08:52:32.258423 1974 rpcz_store.cc:276] 1029 08:52:25.801547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.914474 (+112927us) service_pool.cc:225] Handling call
1029 08:52:32.258090 (+6343616us) ts_tablet_manager.cc:1151] Deleting tablet 2fb9ab76dcfe486e89bbe9391cab56f9
1029 08:52:32.258101 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.258237 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6343354}
I20251029 08:52:32.258955 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1052) took 6324 ms. Trace:
W20251029 08:52:32.259044 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fb9ab76dcfe486e89bbe9391cab56f9 already in progress: creating tablet
I20251029 08:52:32.259073 1974 rpcz_store.cc:276] 1029 08:52:25.934102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:25.934204 (+ 102us) service_pool.cc:225] Handling call
1029 08:52:32.258851 (+6324647us) ts_tablet_manager.cc:1151] Deleting tablet 0c14c63f74f74a18902a17a4d210a90d
1029 08:52:32.258861 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.258939 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6318456}
W20251029 08:52:32.259452 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0c14c63f74f74a18902a17a4d210a90d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c14c63f74f74a18902a17a4d210a90d already in progress: creating tablet
I20251029 08:52:32.259505 1974 tablet_replica.cc:333] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:32.259707 1974 raft_consensus.cc:2243] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:32.260129 1974 raft_consensus.cc:2272] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.260954 1974 ts_tablet_manager.cc:1916] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.268949 753 tablet_service.cc:1505] Processing CreateTablet for tablet ca2cab01089e43b5b8fc20d7cf7a475d (DEFAULT_TABLE table=default.Table_3b11dfccad424815b9e7803c8c2dd90b [id=4284c3adf4b540d3b87ba46eeb0adf61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.274329 889 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca (DEFAULT_TABLE table=default.Table_a6e6d2c56981480d877ea3030fcde485 [id=439ec3c69fdc42369a31ebc725a3053c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.274762 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.277899 416 master-stress-test.cc:577] Killing test thread 1/25
I20251029 08:52:32.282416 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8837d051b71d4dd08557bee10d119ae9 (DEFAULT_TABLE table=default.Table_b0e8b56db4984d038e0b83a4948bd8b5 [id=d5758c6e20a74ccf9a9ebe2774c0a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.282831 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8837d051b71d4dd08557bee10d119ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.283124 895 tablet_service.cc:1505] Processing CreateTablet for tablet ca2cab01089e43b5b8fc20d7cf7a475d (DEFAULT_TABLE table=default.Table_3b11dfccad424815b9e7803c8c2dd90b [id=4284c3adf4b540d3b87ba46eeb0adf61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.283432 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2cab01089e43b5b8fc20d7cf7a475d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.288508 892 tablet_service.cc:1505] Processing CreateTablet for tablet 17f8f822a547460990eaa2544f0a3aff (DEFAULT_TABLE table=default.Table_e3c95dd5d9224e2d8bddacd4dc0be319 [id=0cf6846fdb4947e2acccb6aa7aa3ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.288559 901 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a (DEFAULT_TABLE table=default.Table_7da2a2ba63ee4190bbe7e36d1cac99bf [id=d0ed5cacdac54c3bb391a187751c4c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.288847 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f8f822a547460990eaa2544f0a3aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.289201 887 tablet_service.cc:1505] Processing CreateTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 (DEFAULT_TABLE table=default.Table_e229e75b9314413f8e7b3696ee9e6f55 [id=32ab0b598e01451c8c64bd8c884f5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.289553 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7818ad8e5ec4fca8e3b25019f545dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.290776 888 tablet_service.cc:1505] Processing CreateTablet for tablet 82589b43556c46559f0de4af7e8d9db0 (DEFAULT_TABLE table=default.Table_43aa92e7fba74d91ad0b832e5f5d6592 [id=f8ae6d6bbf1944e5bd793db22dcdfd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.291095 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82589b43556c46559f0de4af7e8d9db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.291307 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.291427 756 tablet_service.cc:1505] Processing CreateTablet for tablet 308b058957c043d08b22a6dd1111c7ff (DEFAULT_TABLE table=default.Table_18732ceb5df84aa88ea943c4d35e6084 [id=dc3fbd651109405fbbb8d9e1a3269087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.291097 896 tablet_service.cc:1505] Processing CreateTablet for tablet 32929763e9b74f3ebb689e6ab5828443 (DEFAULT_TABLE table=default.Table_35877e0584a84248a87841ffcb0cbf94 [id=0e0c97d7ced7490b86425ea2e2e4530c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.163988 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 8a85a1dff3034284a6dcf0e8c3f53297 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a85a1dff3034284a6dcf0e8c3f53297 already in progress: creating tablet
W20251029 08:52:32.291886 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 127887us
I20251029 08:52:32.292129 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32929763e9b74f3ebb689e6ab5828443. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.292121 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]) successfully deleted
W20251029 08:52:32.292335 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 39578ce0411646eea7f6d8dc61490e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39578ce0411646eea7f6d8dc61490e03 already in progress: creating tablet
I20251029 08:52:32.292519 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d306be1d51674c2abc506d248e05a9c7 (table default.Table_09897b5139274e6fb00dd3a14c51db4d [id=6d16b5757b3f494a8e28d2d119d190db]) successfully deleted
W20251029 08:52:32.292708 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet adb59d3ee338465a942e6b4423f0e071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adb59d3ee338465a942e6b4423f0e071 already in progress: creating tablet
I20251029 08:52:32.296151 894 tablet_service.cc:1505] Processing CreateTablet for tablet 33e4694f197b46a1bc2694bf8d91128c (DEFAULT_TABLE table=default.Table_f0938163b61540419aa4d8be600dcc1e [id=1d16258b1e974514be7a21a5782dcbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.296550 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e4694f197b46a1bc2694bf8d91128c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.295320 996 consensus_peers.cc:597] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:32.306610 998 consensus_peers.cc:597] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Couldn't send request to peer 7a8d80098fcb40ada903c3b2bacc2ea0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:32.312103 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_28cea2a6590a436e98fce31e8bf4eff3" } new_table_name: "default.Table_2b4fe6c0772340e68e629dc8b8f2edbd" modify_external_catalogs: true
I20251029 08:52:32.312413 764 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca (DEFAULT_TABLE table=default.Table_a6e6d2c56981480d877ea3030fcde485 [id=439ec3c69fdc42369a31ebc725a3053c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.313256 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:32.314435 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21cfbfbde5541a8976aa9a9b50e8867. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.314724 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_55491b160d0e4e6185be2e0fee2e67a5" } modify_external_catalogs: true
I20251029 08:52:32.314865 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_55491b160d0e4e6185be2e0fee2e67a5" } modify_external_catalogs: true
I20251029 08:52:32.315251 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e4694f197b46a1bc2694bf8d91128c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.316124 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc20228df5849219224d59c1370237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.316577 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8c6a71cd224a1cb7d74e9c2a8f497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.322865 899 tablet_service.cc:1505] Processing CreateTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 (DEFAULT_TABLE table=default.Table_bbe82478d4d64216b1b2fbc3e9c3aade [id=9d70d363be5540ac9c07d20efac755c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.330706 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9bd5a00e42c28c8eadc36795b286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.338557 1183 tablet_bootstrap.cc:492] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.339414 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990e2983ffe4f1ca26f68875416a72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.341019 2554 raft_consensus.cc:493] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:32.341197 2554 raft_consensus.cc:515] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.348713 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66752ba798f54b9790d0750219e4c4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.348755 1178 tablet_bootstrap.cc:492] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.350376 1178 ts_tablet_manager.cc:1403] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.493s user 0.005s sys 0.008s
I20251029 08:52:32.351253 1178 raft_consensus.cc:359] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.351416 1178 raft_consensus.cc:385] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.351475 1178 raft_consensus.cc:740] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.351662 1178 consensus_queue.cc:260] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.352190 1178 ts_tablet_manager.cc:1434] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:32.352259 2869 raft_consensus.cc:493] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.352353 2869 raft_consensus.cc:515] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.352912 2869 leader_election.cc:290] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.354038 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6cacedfdf4cf8b71dba57ad3036ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.354712 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cac145320cf48b4aaf4adb0b686ba8c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.354905 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549826233f6d49fd9bbddb70b6ade0d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.356634 720 leader_election.cc:343] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.357671 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2cab01089e43b5b8fc20d7cf7a475d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.369150 591 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8104bbc33e84e8e8ceecd4e1188e034 consensus state has no leader
I20251029 08:52:32.369580 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e229e75b9314413f8e7b3696ee9e6f55" } modify_external_catalogs: true
I20251029 08:52:32.369727 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_e229e75b9314413f8e7b3696ee9e6f55" } modify_external_catalogs: true
I20251029 08:52:32.378926 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1e2c24e30048abb3ae9fb92171e049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.379894 1178 tablet_bootstrap.cc:492] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
W20251029 08:52:32.380861 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8553f806c78f4617bb8d97cdba2243bf consensus state has no leader
I20251029 08:52:32.381171 590 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39506:
table { table_name: "default.Table_eba74fd181d54950825081b6abd3201a" } new_table_name: "default.Table_0e471dddeaaa4b36912d0cf56846cee9" modify_external_catalogs: true
I20251029 08:52:32.389834 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cac145320cf48b4aaf4adb0b686ba8c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:32.391098 718 leader_election.cc:343] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.391280 718 leader_election.cc:304] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.391796 2920 raft_consensus.cc:2749] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.392068 893 tablet_service.cc:1505] Processing CreateTablet for tablet c21cfbfbde5541a8976aa9a9b50e8867 (DEFAULT_TABLE table=default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.392444 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21cfbfbde5541a8976aa9a9b50e8867. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.393663 2554 leader_election.cc:290] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.393908 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990e2983ffe4f1ca26f68875416a72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.394865 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_7ec9d338fa86438da982cb2b33730cd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.395357 589 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ec9d338fa86438da982cb2b33730cd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 08:52:32.395469 861 leader_election.cc:343] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: d50e2fdfc0e64516a41c90f5a56f4c89
W20251029 08:52:32.395668 861 leader_election.cc:343] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d50e2fdfc0e64516a41c90f5a56f4c89
I20251029 08:52:32.395766 861 leader_election.cc:304] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:32.396270 2554 raft_consensus.cc:2749] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 08:52:32.396611 597 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19414da9e1d34c38ba28a1fc86c43c0d consensus state has no leader
I20251029 08:52:32.397050 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39506:
name: "default.Table_13bd1d85abb34cfa9e1f187cf0b8d566"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 08:52:32.397511 597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13bd1d85abb34cfa9e1f187cf0b8d566 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 08:52:32.397543 416 master-stress-test.cc:579] Killed test thread 1/25
I20251029 08:52:32.397644 416 master-stress-test.cc:577] Killing test thread 2/25
I20251029 08:52:32.397725 416 master-stress-test.cc:579] Killed test thread 2/25
I20251029 08:52:32.397809 416 master-stress-test.cc:577] Killing test thread 3/25
I20251029 08:52:32.397871 416 master-stress-test.cc:579] Killed test thread 3/25
I20251029 08:52:32.397933 416 master-stress-test.cc:577] Killing test thread 4/25
I20251029 08:52:32.398003 416 master-stress-test.cc:579] Killed test thread 4/25
I20251029 08:52:32.398070 416 master-stress-test.cc:577] Killing test thread 5/25
I20251029 08:52:32.398154 416 master-stress-test.cc:579] Killed test thread 5/25
I20251029 08:52:32.398216 416 master-stress-test.cc:577] Killing test thread 6/25
I20251029 08:52:32.398289 416 master-stress-test.cc:579] Killed test thread 6/25
I20251029 08:52:32.398346 416 master-stress-test.cc:577] Killing test thread 7/25
I20251029 08:52:32.398692 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 2fafe9a0731b48b3a60032d94623681d
I20251029 08:52:32.399433 594 master_service.cc:954] ReplaceTablet: received request to replace tablet f8a896408fd142b9b41d606738234ba4 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:32.379482 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1180) took 2584 ms. Trace:
I20251029 08:52:32.400104 1176 rpcz_store.cc:276] 1029 08:52:29.794943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.795006 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:31.798302 (+2003296us) ts_tablet_manager.cc:1151] Deleting tablet 4200db822deb4ace936f561c230ac1fb
1029 08:52:31.798321 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:31.883821 (+ 85500us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.224777 (+340956us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.379451 (+154674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1985224}
I20251029 08:52:32.400779 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1187) took 2468 ms. Trace:
I20251029 08:52:32.400867 1176 rpcz_store.cc:276] 1029 08:52:29.932005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.932083 (+ 78us) service_pool.cc:225] Handling call
1029 08:52:32.400636 (+2468553us) ts_tablet_manager.cc:1151] Deleting tablet ea40ed38c0ba4d2f9045188af226ced5
1029 08:52:32.400650 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.400759 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2468343}
I20251029 08:52:32.401275 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1188) took 2467 ms. Trace:
I20251029 08:52:32.401342 1176 rpcz_store.cc:276] 1029 08:52:29.934133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.934196 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:32.401185 (+2466989us) ts_tablet_manager.cc:1151] Deleting tablet 307262da8888461ebddaaf33a31be8b8
1029 08:52:32.401194 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.401263 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466819}
I20251029 08:52:32.401743 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1189) took 2465 ms. Trace:
I20251029 08:52:32.401809 1176 rpcz_store.cc:276] 1029 08:52:29.936271 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.936320 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:32.401655 (+2465335us) ts_tablet_manager.cc:1151] Deleting tablet 4675f0aec6a14807afd3094780d60c72
1029 08:52:32.401665 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.401731 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465166}
I20251029 08:52:32.402179 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1190) took 2449 ms. Trace:
I20251029 08:52:32.402245 1176 rpcz_store.cc:276] 1029 08:52:29.952643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.952699 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:32.402083 (+2449384us) ts_tablet_manager.cc:1151] Deleting tablet 4524a248859448c5b789c7ae82435705
1029 08:52:32.402090 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.402164 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2449148}
I20251029 08:52:32.402590 1176 tablet_replica.cc:333] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:32.402766 1176 raft_consensus.cc:2243] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:32.402912 1176 raft_consensus.cc:2272] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.403424 1176 ts_tablet_manager.cc:1916] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:32.403527 592 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e9a0066c04948c6af4f77564e66832a consensus state has no leader
I20251029 08:52:32.403770 2359 raft_consensus.cc:493] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.403927 2359 raft_consensus.cc:515] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.404830 2359 leader_election.cc:290] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.405257 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3a2fab467348099a4a1fec74eb5e65" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:32.405786 996 leader_election.cc:343] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.407119 2731 raft_consensus.cc:493] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.407279 2731 raft_consensus.cc:515] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:32.408591 861 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: e3c24c0d32f74435bd9d28f2415e2c47
W20251029 08:52:32.409279 998 leader_election.cc:343] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 4e3a2fab467348099a4a1fec74eb5e65
I20251029 08:52:32.409415 998 leader_election.cc:304] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.409920 2359 raft_consensus.cc:2749] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:32.411103 832 tablet.cc:2378] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:32.411653 886 tablet_service.cc:1505] Processing CreateTablet for tablet e6403e337ac3477082ba730c5312e591 (DEFAULT_TABLE table=default.Table_532d58c219f94321bf5460c99c389e1e [id=bc4d0acb083046c6b72715c48235a72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.412009 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6403e337ac3477082ba730c5312e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.413496 2359 raft_consensus.cc:493] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.413612 2359 raft_consensus.cc:515] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.414137 2359 leader_election.cc:290] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.414534 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.414963 887 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd6c279f9f4d7282d66e2e4101b668 (DEFAULT_TABLE table=default.Table_e52a0b50fc624679bc5d91a8cf424a20 [id=547c00ece3cf465a87d1e275010e78de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.415271 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd6c279f9f4d7282d66e2e4101b668. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.415792 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:32.415949 778 raft_consensus.cc:2468] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:32.416370 998 leader_election.cc:304] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:32.416779 2359 raft_consensus.cc:2804] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:32.416846 2359 raft_consensus.cc:493] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:32.416894 2359 raft_consensus.cc:3060] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.419317 2359 raft_consensus.cc:515] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.419952 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc87eea9f3849ac85ccd7e0a5d8f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.420089 2359 leader_election.cc:290] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.420742 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:32.420957 778 raft_consensus.cc:3060] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.423394 1178 tablet_bootstrap.cc:654] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.423799 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b058957c043d08b22a6dd1111c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.439141 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:38031, user_credentials={real_user=slave}} blocked reactor thread for 144954us
I20251029 08:52:32.440871 889 tablet_service.cc:1505] Processing CreateTablet for tablet 19414da9e1d34c38ba28a1fc86c43c0d (DEFAULT_TABLE table=default.Table_ec5782dfc71a40e5870d6fc9a8e51780 [id=1ba323d0b8eb46a082394b69e7c3efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.441857 2359 consensus_queue.cc:1048] T 66329e1e548949cc9d59097953288375 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 08:52:32.447471 996 leader_election.cc:343] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.447985 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9fe3e9a507441b8ba5953c0120289c3" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:32.448383 2731 leader_election.cc:290] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:32.448448 996 leader_election.cc:343] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.453331 900 tablet_service.cc:1505] Processing CreateTablet for tablet b3de9bd5a00e42c28c8eadc36795b286 (DEFAULT_TABLE table=default.Table_4189d781b1c24ebaae9225c1c511ec73 [id=b6b16a48311746539375abadb39dde2e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.454939 861 leader_election.cc:343] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e3c24c0d32f74435bd9d28f2415e2c47
I20251029 08:52:32.455053 861 leader_election.cc:304] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:32.455588 2731 raft_consensus.cc:2749] T e3c24c0d32f74435bd9d28f2415e2c47 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.456604 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19414da9e1d34c38ba28a1fc86c43c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.460156 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9bd5a00e42c28c8eadc36795b286. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.461549 590 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5062ddafd784a428158aa537955eddd consensus state has no leader
I20251029 08:52:32.462931 1178 tablet_bootstrap.cc:492] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.463070 1178 ts_tablet_manager.cc:1403] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.000s
I20251029 08:52:32.463769 901 tablet_service.cc:1505] Processing CreateTablet for tablet ebc20228df5849219224d59c1370237b (DEFAULT_TABLE table=default.Table_cbee24e26857417db3a2cb72aca81cb5 [id=502c1023eff24760ac2629849642b5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.465169 1178 raft_consensus.cc:359] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.465348 1178 raft_consensus.cc:385] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.465394 1178 raft_consensus.cc:740] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.465632 1178 consensus_queue.cc:260] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.466120 1178 ts_tablet_manager.cc:1434] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 08:52:32.466548 1178 tablet_bootstrap.cc:492] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:32.468881 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc20228df5849219224d59c1370237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.469413 1176 ts_tablet_manager.cc:1929] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:32.469522 1176 log.cc:1199] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d5bb0fd4c62d47a2b10719ebd55851c0
I20251029 08:52:32.469852 1176 ts_tablet_manager.cc:1950] T d5bb0fd4c62d47a2b10719ebd55851c0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:32.472100 416 master-stress-test.cc:579] Killed test thread 7/25
I20251029 08:52:32.472215 416 master-stress-test.cc:577] Killing test thread 8/25
I20251029 08:52:32.472295 416 master-stress-test.cc:579] Killed test thread 8/25
I20251029 08:52:32.472363 416 master-stress-test.cc:577] Killing test thread 9/25
I20251029 08:52:32.472429 416 master-stress-test.cc:579] Killed test thread 9/25
I20251029 08:52:32.472494 416 master-stress-test.cc:577] Killing test thread 10/25
I20251029 08:52:32.472571 416 master-stress-test.cc:579] Killed test thread 10/25
I20251029 08:52:32.472643 416 master-stress-test.cc:577] Killing test thread 11/25
I20251029 08:52:32.472697 416 master-stress-test.cc:579] Killed test thread 11/25
I20251029 08:52:32.472760 416 master-stress-test.cc:577] Killing test thread 12/25
I20251029 08:52:32.472829 416 master-stress-test.cc:579] Killed test thread 12/25
I20251029 08:52:32.472904 416 master-stress-test.cc:577] Killing test thread 13/25
I20251029 08:52:32.472985 416 master-stress-test.cc:579] Killed test thread 13/25
I20251029 08:52:32.473059 416 master-stress-test.cc:577] Killing test thread 14/25
I20251029 08:52:32.473136 416 master-stress-test.cc:579] Killed test thread 14/25
I20251029 08:52:32.473213 416 master-stress-test.cc:577] Killing test thread 15/25
I20251029 08:52:32.473289 416 master-stress-test.cc:579] Killed test thread 15/25
I20251029 08:52:32.473362 416 master-stress-test.cc:577] Killing test thread 16/25
I20251029 08:52:32.475059 1178 tablet_bootstrap.cc:654] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.478051 1177 tablet_bootstrap.cc:492] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:32.478114 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:32.478402 591 catalog_manager.cc:5649] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:32.478829 591 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:32.479472 1177 ts_tablet_manager.cc:1403] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.289s user 0.014s sys 0.026s
I20251029 08:52:32.489012 596 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 1b3ab5946c574171b66f206c85720f79
I20251029 08:52:32.491990 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 82589b43556c46559f0de4af7e8d9db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.492226 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ac09bb4502e440368633bbc90e9ad2cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.492362 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a17142ffce3e4c359e715ef63747c40d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.492460 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d010353b251140e99ce2df996f965991 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.492554 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fafe9a0731b48b3a60032d94623681d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.492904 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1b3ab5946c574171b66f206c85720f79 (DEFAULT_TABLE table=default.Table_b3c1dbf9db424137b9c0cec53bb32311 [id=24b1d52443014c5fa7f4b8ce5c7b97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.493273 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3ab5946c574171b66f206c85720f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.494964 593 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet c7818ad8e5ec4fca8e3b25019f545dd8
I20251029 08:52:32.495333 595 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet d444a5d489434696ae178c6d650bf8d5
I20251029 08:52:32.496012 594 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet f8a896408fd142b9b41d606738234ba4
I20251029 08:52:32.496575 594 catalog_manager.cc:6719] ReplaceTablet: tablet f8a896408fd142b9b41d606738234ba4 deleted and replaced by tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d
I20251029 08:52:32.496194 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet cb1e2c24e30048abb3ae9fb92171e049
I20251029 08:52:32.497790 416 master-stress-test.cc:579] Killed test thread 16/25
I20251029 08:52:32.497900 416 master-stress-test.cc:577] Killing test thread 17/25
I20251029 08:52:32.497988 416 master-stress-test.cc:579] Killed test thread 17/25
I20251029 08:52:32.498066 416 master-stress-test.cc:577] Killing test thread 18/25
I20251029 08:52:32.498157 416 master-stress-test.cc:579] Killed test thread 18/25
I20251029 08:52:32.498226 416 master-stress-test.cc:577] Killing test thread 19/25
I20251029 08:52:32.498292 416 master-stress-test.cc:579] Killed test thread 19/25
I20251029 08:52:32.498359 416 master-stress-test.cc:577] Killing test thread 20/25
I20251029 08:52:32.498440 416 master-stress-test.cc:579] Killed test thread 20/25
I20251029 08:52:32.498520 416 master-stress-test.cc:577] Killing test thread 21/25
I20251029 08:52:32.499912 905 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6402327d0548d9ba043c6117afefcf (DEFAULT_TABLE table=default.Table_ad961d1397114acab3368b402e9fd83f [id=f966fe0470e04d74bbedadc9c2640e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.500371 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6402327d0548d9ba043c6117afefcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.508812 904 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b3ab5946c574171b66f206c85720f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.509167 904 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21abab863b439b8ce1046b6d90ddc8 (DEFAULT_TABLE table=default.Table_c7cec907d0c54aa297ab2041445ef20e [id=08101bd8a9944718a7cad2877e63aa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.509510 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21abab863b439b8ce1046b6d90ddc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.510521 1183 ts_tablet_manager.cc:1403] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.389s user 0.019s sys 0.002s
I20251029 08:52:32.511266 1183 raft_consensus.cc:359] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.512089 1183 raft_consensus.cc:385] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.512099 898 tablet_service.cc:1552] Processing DeleteTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.512212 1183 raft_consensus.cc:740] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.512270 898 tablet_service.cc:1552] Processing DeleteTablet for tablet d444a5d489434696ae178c6d650bf8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.512416 898 tablet_service.cc:1552] Processing DeleteTablet for tablet f8a896408fd142b9b41d606738234ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.512557 898 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.512864 898 tablet_service.cc:1505] Processing CreateTablet for tablet afb32905315b4077926d028eaa404967 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.513211 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb32905315b4077926d028eaa404967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.520085 1183 consensus_queue.cc:260] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.521034 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:32.525841 896 tablet_service.cc:1505] Processing CreateTablet for tablet 70ba0bb50d44426b90fa0d70453bc1a6 (DEFAULT_TABLE table=default.Table_98334d1800134673a0ee67105e125d80 [id=29244ca64c434a0596f329633738be39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.530301 1974 ts_tablet_manager.cc:1929] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:32.532693 1974 log.cc:1199] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/39fc9612689b46d498d6092bd552da4e
I20251029 08:52:32.534399 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:32.534972 891 tablet_service.cc:1505] Processing CreateTablet for tablet d8df074547624d449c46dba235bf7d7a (DEFAULT_TABLE table=default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.535380 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8df074547624d449c46dba235bf7d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.538620 890 tablet_service.cc:1505] Processing CreateTablet for tablet 0c7e0f233b6b4f65843915909278d28f (DEFAULT_TABLE table=default.Table_b902ca54a7614b7086c649649b3a6646 [id=acde8367aa004830b6e721fed5321cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.539029 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7e0f233b6b4f65843915909278d28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.540344 1183 ts_tablet_manager.cc:1434] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20251029 08:52:32.542716 1974 ts_tablet_manager.cc:1950] T 39fc9612689b46d498d6092bd552da4e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:32.543047 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1062) took 6422 ms. Trace:
I20251029 08:52:32.544440 1974 rpcz_store.cc:276] 1029 08:52:26.120542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.120597 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:32.259452 (+6138855us) ts_tablet_manager.cc:1151] Deleting tablet 39fc9612689b46d498d6092bd552da4e
1029 08:52:32.259461 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.451126 (+191665us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.530186 (+ 79060us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.543033 (+ 12847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6138660}
I20251029 08:52:32.544854 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 39fc9612689b46d498d6092bd552da4e (table default.Table_5f9765183d954404b485e1a99c508874 [id=71e2fbee1f464441a7db68398090f695]) successfully deleted
I20251029 08:52:32.544955 1974 tablet_replica.cc:333] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:32.545151 1974 raft_consensus.cc:2243] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:32.545491 1974 raft_consensus.cc:2272] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.547241 1183 tablet_bootstrap.cc:492] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.547982 1178 tablet_bootstrap.cc:492] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.548178 1178 ts_tablet_manager.cc:1403] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20251029 08:52:32.549367 1178 raft_consensus.cc:359] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.549599 1178 raft_consensus.cc:385] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.549688 1178 raft_consensus.cc:740] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.550096 1178 consensus_queue.cc:260] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.556172 1178 ts_tablet_manager.cc:1434] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 08:52:32.556548 1178 tablet_bootstrap.cc:492] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:32.572975 1178 tablet_bootstrap.cc:654] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.573902 1183 tablet_bootstrap.cc:654] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.576400 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 82589b43556c46559f0de4af7e8d9db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.576452 764 tablet_service.cc:1552] Processing DeleteTablet for tablet 82589b43556c46559f0de4af7e8d9db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.577003 764 tablet_service.cc:1552] Processing DeleteTablet for tablet ac09bb4502e440368633bbc90e9ad2cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.577538 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet ac09bb4502e440368633bbc90e9ad2cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.578590 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 4200db822deb4ace936f561c230ac1fb with delete_type TABLET_DATA_DELETED (replaced by tablet 5498826a9afd4fd5a6b88b0a465fff40) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.579082 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fb9ab76dcfe486e89bbe9391cab56f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.581831 1178 tablet_bootstrap.cc:492] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.581949 1178 ts_tablet_manager.cc:1403] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.006s
I20251029 08:52:32.582929 1178 raft_consensus.cc:359] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.583050 2359 raft_consensus.cc:493] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.583098 1178 raft_consensus.cc:385] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.583153 1178 raft_consensus.cc:740] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.583326 1178 consensus_queue.cc:260] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.583362 2359 raft_consensus.cc:515] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.583595 2920 raft_consensus.cc:493] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.583693 2920 raft_consensus.cc:515] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.584183 2920 leader_election.cc:290] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.584213 2359 leader_election.cc:290] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.585079 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2059cb9767a4fdda690d91322511cf4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:32.586284 998 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b8f2b34642704cd6bb966b6dcd494d51
W20251029 08:52:32.588105 720 leader_election.cc:343] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.595631 778 raft_consensus.cc:2468] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:32.596340 998 leader_election.cc:304] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.597018 2896 raft_consensus.cc:2804] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:32.597214 2896 raft_consensus.cc:697] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:32.597591 2896 consensus_queue.cc:237] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.597880 1178 ts_tablet_manager.cc:1434] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251029 08:52:32.598066 2939 tablet.cc:1722] T 2dec33b502aa4916b89b878462de1ab1 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:32.603453 1178 tablet_bootstrap.cc:492] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:32.605132 1178 tablet_bootstrap.cc:654] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.598886 2938 tablet_replica.cc:333] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:32.608291 2938 raft_consensus.cc:2243] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:32.608628 2938 pending_rounds.cc:70] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:32.608731 2938 pending_rounds.cc:77] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037688722808832 op_type: NO_OP noop_request { }
I20251029 08:52:32.608764 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a69beee4032f447eaa694d237cb0e4e3 (table default.Table_bb583028431740f4afd61a171a14298e [id=d687628169ff44f5b2cb137a0bc8e325]). Expected version 2 got 0
I20251029 08:52:32.608887 2938 raft_consensus.cc:2889] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:32.609020 2938 raft_consensus.cc:2272] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.609624 2938 ts_tablet_manager.cc:1916] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.616370 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet a69beee4032f447eaa694d237cb0e4e3 (table default.Table_bb583028431740f4afd61a171a14298e [id=d687628169ff44f5b2cb137a0bc8e325]). Expected version 2 got 0
I20251029 08:52:32.617245 2940 tablet.cc:1722] T 2dec33b502aa4916b89b878462de1ab1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:32.627192 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.632932 1183 tablet_bootstrap.cc:492] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.633082 1183 ts_tablet_manager.cc:1403] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20251029 08:52:32.633900 1183 raft_consensus.cc:359] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:32.634117 1183 raft_consensus.cc:385] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.636044 588 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4] alter complete (version 2)
I20251029 08:52:32.550450 895 tablet_service.cc:1505] Processing CreateTablet for tablet b0648a22ba8e4124ad5b8920f941aa49 (DEFAULT_TABLE table=default.Table_6eb0f7c99526409c86c08f5dfbe63e58 [id=57081dd11f174114a078b5581b9f37e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.551465 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba0bb50d44426b90fa0d70453bc1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.557885 893 tablet_service.cc:1505] Processing CreateTablet for tablet 4c878070c3294dabb598598da2648161 (DEFAULT_TABLE table=default.Table_98d653f845be48e7bb3ccd4d1000e926 [id=5b3d4957444e4c3293adc5abdea133fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.647647 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c878070c3294dabb598598da2648161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.647617 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a17142ffce3e4c359e715ef63747c40d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.647831 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d010353b251140e99ce2df996f965991 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.648403 2920 raft_consensus.cc:493] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.648533 2920 raft_consensus.cc:515] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.649112 2920 leader_election.cc:290] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:32.650552 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f67e7b28b4240618e1817bb47d5633d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:32.651237 718 leader_election.cc:343] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.651801 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.654021 757 tablet_service.cc:1505] Processing CreateTablet for tablet 1b3ab5946c574171b66f206c85720f79 (DEFAULT_TABLE table=default.Table_b3c1dbf9db424137b9c0cec53bb32311 [id=24b1d52443014c5fa7f4b8ce5c7b97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.654332 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fafe9a0731b48b3a60032d94623681d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.654558 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 308da528313044929b66040d6659f774 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.655562 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3ab5946c574171b66f206c85720f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.656183 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0648a22ba8e4124ad5b8920f941aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.560384 899 tablet_service.cc:1505] Processing CreateTablet for tablet b4c14fc80de147cba1f1e41833bafd2f (DEFAULT_TABLE table=default.Table_66e64babc1194427be120879d68f8617 [id=bf2aaf5dad57418cb33c73de19a4d5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.659515 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c14fc80de147cba1f1e41833bafd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.675864 751 tablet_service.cc:1505] Processing CreateTablet for tablet afb32905315b4077926d028eaa404967 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.676498 757 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6402327d0548d9ba043c6117afefcf (DEFAULT_TABLE table=default.Table_ad961d1397114acab3368b402e9fd83f [id=f966fe0470e04d74bbedadc9c2640e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.677019 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21abab863b439b8ce1046b6d90ddc8 (DEFAULT_TABLE table=default.Table_c7cec907d0c54aa297ab2041445ef20e [id=08101bd8a9944718a7cad2877e63aa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.568481 901 tablet_service.cc:1505] Processing CreateTablet for tablet e24772878f6e457aa270edcf44d47438 (DEFAULT_TABLE table=default.Table_6944e94b47b34496aa62902b739cfa85 [id=f20a9239ae9144c0974cae391763b3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.584416 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f2b34642704cd6bb966b6dcd494d51" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.585963 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2059cb9767a4fdda690d91322511cf4" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:32.677416 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b3ab5946c574171b66f206c85720f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.678241 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24772878f6e457aa270edcf44d47438. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.678149 718 leader_election.cc:343] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.678606 718 leader_election.cc:304] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.678900 2920 raft_consensus.cc:2749] T a2059cb9767a4fdda690d91322511cf4 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.682173 1974 ts_tablet_manager.cc:1916] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.685267 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb32905315b4077926d028eaa404967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.689805 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6402327d0548d9ba043c6117afefcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.693778 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21abab863b439b8ce1046b6d90ddc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.641418 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet a17142ffce3e4c359e715ef63747c40d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.642192 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet d010353b251140e99ce2df996f965991 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.649611 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f67e7b28b4240618e1817bb47d5633d" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:32.704406 1183 raft_consensus.cc:740] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.653684 1050 tablet_service.cc:1505] Processing CreateTablet for tablet 1b3ab5946c574171b66f206c85720f79 (DEFAULT_TABLE table=default.Table_b3c1dbf9db424137b9c0cec53bb32311 [id=24b1d52443014c5fa7f4b8ce5c7b97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.704958 1050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b3ab5946c574171b66f206c85720f79. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.659109 1119 tablet.cc:2378] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 08:52:32.708230 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 9f8348b6234e4dbe8bdb58edad34ea7b from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:32.660769 2938 ts_tablet_manager.cc:1929] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:32.709506 2938 log.cc:1199] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4200db822deb4ace936f561c230ac1fb
I20251029 08:52:32.709944 2938 ts_tablet_manager.cc:1950] T 4200db822deb4ace936f561c230ac1fb P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:32.711002 2938 tablet_replica.cc:333] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:32.711177 2938 raft_consensus.cc:2243] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:32.711330 2938 raft_consensus.cc:2272] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.711867 2938 ts_tablet_manager.cc:1916] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.672959 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6402327d0548d9ba043c6117afefcf (DEFAULT_TABLE table=default.Table_ad961d1397114acab3368b402e9fd83f [id=f966fe0470e04d74bbedadc9c2640e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.712345 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6402327d0548d9ba043c6117afefcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.683259 1177 raft_consensus.cc:359] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:32.712455 1177 raft_consensus.cc:385] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.712543 1177 raft_consensus.cc:740] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.712905 764 tablet_service.cc:1505] Processing CreateTablet for tablet 70ba0bb50d44426b90fa0d70453bc1a6 (DEFAULT_TABLE table=default.Table_98334d1800134673a0ee67105e125d80 [id=29244ca64c434a0596f329633738be39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.713285 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba0bb50d44426b90fa0d70453bc1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.673394 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b3ab5946c574171b66f206c85720f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:31 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.674252 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21abab863b439b8ce1046b6d90ddc8 (DEFAULT_TABLE table=default.Table_c7cec907d0c54aa297ab2041445ef20e [id=08101bd8a9944718a7cad2877e63aa30]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.677759 996 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.716240 996 leader_election.cc:304] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.716414 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21abab863b439b8ce1046b6d90ddc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.717849 720 leader_election.cc:343] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:32.717898 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 0c7e0f233b6b4f65843915909278d28f (DEFAULT_TABLE table=default.Table_b902ca54a7614b7086c649649b3a6646 [id=acde8367aa004830b6e721fed5321cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.717978 720 leader_election.cc:304] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:32.718081 1045 tablet_service.cc:1552] Processing DeleteTablet for tablet f8a896408fd142b9b41d606738234ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.718247 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7e0f233b6b4f65843915909278d28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.718286 1046 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.718389 2920 raft_consensus.cc:2749] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.718604 2707 raft_consensus.cc:2749] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.719067 752 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1e2c24e30048abb3ae9fb92171e049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.675415 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.652839 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fafe9a0731b48b3a60032d94623681d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.720000 1036 tablet_service.cc:1505] Processing CreateTablet for tablet afb32905315b4077926d028eaa404967 (DEFAULT_TABLE table=default.Table_580fb63e5fcb4335990145e4f4fe128c [id=e1bff87c9b064ea68643f9b6306b1354]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.720357 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb32905315b4077926d028eaa404967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.720466 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d8df074547624d449c46dba235bf7d7a (DEFAULT_TABLE table=default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.720777 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8df074547624d449c46dba235bf7d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.724176 1177 consensus_queue.cc:260] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:32.724268 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 70ba0bb50d44426b90fa0d70453bc1a6 (DEFAULT_TABLE table=default.Table_98334d1800134673a0ee67105e125d80 [id=29244ca64c434a0596f329633738be39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.724592 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ba0bb50d44426b90fa0d70453bc1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.725023 761 tablet_service.cc:1552] Processing DeleteTablet for tablet f8a896408fd142b9b41d606738234ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.728258 1043 tablet_service.cc:1505] Processing CreateTablet for tablet b0648a22ba8e4124ad5b8920f941aa49 (DEFAULT_TABLE table=default.Table_6eb0f7c99526409c86c08f5dfbe63e58 [id=57081dd11f174114a078b5581b9f37e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.728572 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0648a22ba8e4124ad5b8920f941aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.729343 1177 ts_tablet_manager.cc:1434] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.250s user 0.001s sys 0.001s
I20251029 08:52:32.729713 1177 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:32.729763 761 tablet_service.cc:1505] Processing CreateTablet for tablet d8df074547624d449c46dba235bf7d7a (DEFAULT_TABLE table=default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.730116 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8df074547624d449c46dba235bf7d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.731644 1177 tablet_bootstrap.cc:654] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.732151 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f67e7b28b4240618e1817bb47d5633d with delete_type TABLET_DATA_DELETED (replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.732295 1049 tablet_service.cc:1505] Processing CreateTablet for tablet 4c878070c3294dabb598598da2648161 (DEFAULT_TABLE table=default.Table_98d653f845be48e7bb3ccd4d1000e926 [id=5b3d4957444e4c3293adc5abdea133fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.732409 752 tablet_service.cc:1505] Processing CreateTablet for tablet 0c7e0f233b6b4f65843915909278d28f (DEFAULT_TABLE table=default.Table_b902ca54a7614b7086c649649b3a6646 [id=acde8367aa004830b6e721fed5321cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.732620 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c878070c3294dabb598598da2648161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.732734 758 tablet_service.cc:1505] Processing CreateTablet for tablet 4c878070c3294dabb598598da2648161 (DEFAULT_TABLE table=default.Table_98d653f845be48e7bb3ccd4d1000e926 [id=5b3d4957444e4c3293adc5abdea133fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.732860 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7e0f233b6b4f65843915909278d28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.733036 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c878070c3294dabb598598da2648161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.736284 1041 tablet_service.cc:1505] Processing CreateTablet for tablet b4c14fc80de147cba1f1e41833bafd2f (DEFAULT_TABLE table=default.Table_66e64babc1194427be120879d68f8617 [id=bf2aaf5dad57418cb33c73de19a4d5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.736622 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c14fc80de147cba1f1e41833bafd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.737151 750 tablet_service.cc:1505] Processing CreateTablet for tablet b4c14fc80de147cba1f1e41833bafd2f (DEFAULT_TABLE table=default.Table_66e64babc1194427be120879d68f8617 [id=bf2aaf5dad57418cb33c73de19a4d5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.737468 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c14fc80de147cba1f1e41833bafd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.740264 1044 tablet_service.cc:1505] Processing CreateTablet for tablet e24772878f6e457aa270edcf44d47438 (DEFAULT_TABLE table=default.Table_6944e94b47b34496aa62902b739cfa85 [id=f20a9239ae9144c0974cae391763b3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.740595 1044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24772878f6e457aa270edcf44d47438. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.741276 745 tablet_service.cc:1505] Processing CreateTablet for tablet e24772878f6e457aa270edcf44d47438 (DEFAULT_TABLE table=default.Table_6944e94b47b34496aa62902b739cfa85 [id=f20a9239ae9144c0974cae391763b3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.741614 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24772878f6e457aa270edcf44d47438. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.743908 759 tablet_service.cc:1505] Processing CreateTablet for tablet b0648a22ba8e4124ad5b8920f941aa49 (DEFAULT_TABLE table=default.Table_6eb0f7c99526409c86c08f5dfbe63e58 [id=57081dd11f174114a078b5581b9f37e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.744266 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0648a22ba8e4124ad5b8920f941aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.744974 2554 raft_consensus.cc:493] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:32.745112 2554 raft_consensus.cc:515] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.745266 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f36641b7086430a89380056e04b8aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.745834 2554 leader_election.cc:290] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:32.746558 861 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: aa0ef904c2fb4dfeadbf56ae659c4e20
W20251029 08:52:32.747359 861 leader_election.cc:343] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:32.747416 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet d444a5d489434696ae178c6d650bf8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.747454 861 leader_election.cc:304] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:32.747681 2554 raft_consensus.cc:2749] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 08:52:32.749061 756 tablet_service.cc:1552] Processing DeleteTablet for tablet c7818ad8e5ec4fca8e3b25019f545dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.749333 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d444a5d489434696ae178c6d650bf8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.704819 1183 consensus_queue.cc:260] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:32.756768 1183 ts_tablet_manager.cc:1434] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.124s user 0.002s sys 0.000s
I20251029 08:52:32.758847 1183 tablet_bootstrap.cc:492] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.760738 2944 tablet.cc:1722] T 2fafe9a0731b48b3a60032d94623681d P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:32.761178 1183 tablet_bootstrap.cc:654] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.777999 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1192) took 2797 ms. Trace:
I20251029 08:52:32.778132 1176 rpcz_store.cc:276] 1029 08:52:29.980376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.980441 (+ 65us) service_pool.cc:225] Handling call
1029 08:52:32.402549 (+2422108us) ts_tablet_manager.cc:1151] Deleting tablet d5bb0fd4c62d47a2b10719ebd55851c0
1029 08:52:32.402557 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.466327 (+ 63770us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.469329 (+ 3002us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.777969 (+308640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421879}
I20251029 08:52:32.778549 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1193) took 2761 ms. Trace:
I20251029 08:52:32.778637 1176 rpcz_store.cc:276] 1029 08:52:30.017170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.017227 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:32.778419 (+2761192us) ts_tablet_manager.cc:1151] Deleting tablet f2c18e4c41714ba38bdd2e208a5e761e
1029 08:52:32.778429 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.778530 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2761005}
I20251029 08:52:32.778970 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1196) took 2591 ms. Trace:
I20251029 08:52:32.779048 1176 rpcz_store.cc:276] 1029 08:52:30.188015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.188199 (+ 184us) service_pool.cc:225] Handling call
1029 08:52:32.778873 (+2590674us) ts_tablet_manager.cc:1151] Deleting tablet 3ee83f5526f74ba2a344edd97db94050
1029 08:52:32.778883 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.778956 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2590351}
I20251029 08:52:32.779310 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1197) took 2588 ms. Trace:
I20251029 08:52:32.779376 1176 rpcz_store.cc:276] 1029 08:52:30.190510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.190558 (+ 48us) service_pool.cc:225] Handling call
1029 08:52:32.779230 (+2588672us) ts_tablet_manager.cc:1151] Deleting tablet e27701fbf5b44e90ba33bed99fdc4d9b
1029 08:52:32.779238 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.779300 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588460}
I20251029 08:52:32.779626 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1198) took 2587 ms. Trace:
I20251029 08:52:32.779706 1176 rpcz_store.cc:276] 1029 08:52:30.192564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.192674 (+ 110us) service_pool.cc:225] Handling call
1029 08:52:32.779538 (+2586864us) ts_tablet_manager.cc:1151] Deleting tablet 9e44e56554574d5abb67d08d88218152
1029 08:52:32.779545 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.779611 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2585969}
I20251029 08:52:32.780879 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 4200db822deb4ace936f561c230ac1fb (table default.Table_b6d28fc5fdc14b10b3436923d1b86bc1 [id=572a15f289894adb908547d8f011765a]) successfully deleted
W20251029 08:52:32.781134 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ea40ed38c0ba4d2f9045188af226ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea40ed38c0ba4d2f9045188af226ced5 already in progress: creating tablet
I20251029 08:52:32.779963 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1199) took 2497 ms. Trace:
W20251029 08:52:32.781322 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 307262da8888461ebddaaf33a31be8b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 307262da8888461ebddaaf33a31be8b8 already in progress: creating tablet
I20251029 08:52:32.781309 1176 rpcz_store.cc:276] 1029 08:52:30.282774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.291600 (+ 8826us) service_pool.cc:225] Handling call
1029 08:52:32.779886 (+2488286us) ts_tablet_manager.cc:1151] Deleting tablet 0adba75cc7d74b1497de9af7ad7ccdd0
1029 08:52:32.779893 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.779950 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2488068}
I20251029 08:52:32.781582 1176 tablet_replica.cc:333] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:32.781787 1176 raft_consensus.cc:2243] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:32.781953 1176 raft_consensus.cc:2272] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.782514 1176 ts_tablet_manager.cc:1916] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:32.792598 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4675f0aec6a14807afd3094780d60c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4675f0aec6a14807afd3094780d60c72 already in progress: creating tablet
W20251029 08:52:32.792873 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4524a248859448c5b789c7ae82435705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4524a248859448c5b789c7ae82435705 already in progress: creating tablet
I20251029 08:52:32.793563 2938 ts_tablet_manager.cc:1929] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:32.806149 2938 log.cc:1199] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2fb9ab76dcfe486e89bbe9391cab56f9
I20251029 08:52:32.806638 2938 ts_tablet_manager.cc:1950] T 2fb9ab76dcfe486e89bbe9391cab56f9 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:32.810906 2938 tablet_replica.cc:333] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:32.811858 1176 ts_tablet_manager.cc:1929] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:32.812258 2938 raft_consensus.cc:2243] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:32.818552 2938 raft_consensus.cc:2272] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.819906 2938 ts_tablet_manager.cc:1916] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.820150 1176 log.cc:1199] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/31bed64a27164baa8177b87b54b5947b
I20251029 08:52:32.820557 1176 ts_tablet_manager.cc:1950] T 31bed64a27164baa8177b87b54b5947b P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:32.821382 2940 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 7a8d80098fcb40ada903c3b2bacc2ea0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:32.820286 2939 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 730d0f5b7abf43a293b7df13d01ce4ee: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:32.823688 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1205) took 2313 ms. Trace:
I20251029 08:52:32.823889 1176 rpcz_store.cc:276] 1029 08:52:30.510664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.510749 (+ 85us) service_pool.cc:225] Handling call
1029 08:52:32.781533 (+2270784us) ts_tablet_manager.cc:1151] Deleting tablet 31bed64a27164baa8177b87b54b5947b
1029 08:52:32.781544 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.798740 (+ 17196us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.811736 (+ 12996us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.823664 (+ 11928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270607}
I20251029 08:52:32.824359 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1209) took 2159 ms. Trace:
I20251029 08:52:32.824432 1176 rpcz_store.cc:276] 1029 08:52:30.665401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.665476 (+ 75us) service_pool.cc:225] Handling call
1029 08:52:32.824200 (+2158724us) ts_tablet_manager.cc:1151] Deleting tablet 7492195a41664ef69554d8a6cb148e21
1029 08:52:32.824212 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.824343 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059162}
I20251029 08:52:32.824764 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1213) took 2153 ms. Trace:
I20251029 08:52:32.824836 1176 rpcz_store.cc:276] 1029 08:52:30.671770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.671975 (+ 205us) service_pool.cc:225] Handling call
1029 08:52:32.824660 (+2152685us) ts_tablet_manager.cc:1151] Deleting tablet 4dec9fdf1ae94ae687f30f89096ce89b
1029 08:52:32.824670 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.824749 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045870}
I20251029 08:52:32.825102 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1214) took 2111 ms. Trace:
I20251029 08:52:32.825162 1176 rpcz_store.cc:276] 1029 08:52:30.713881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.713928 (+ 47us) service_pool.cc:225] Handling call
1029 08:52:32.825015 (+2111087us) ts_tablet_manager.cc:1151] Deleting tablet 07bf5d693be240f18bacccc81a6ea5d4
1029 08:52:32.825023 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.825090 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2042512}
I20251029 08:52:32.825445 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1224) took 2006 ms. Trace:
I20251029 08:52:32.825508 1176 rpcz_store.cc:276] 1029 08:52:30.818946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.819002 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:32.825335 (+2006333us) ts_tablet_manager.cc:1151] Deleting tablet 6dc6c8f9ee9840c08c9436ab516a62d7
1029 08:52:32.825345 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.825408 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006156}
I20251029 08:52:32.801551 1183 tablet_bootstrap.cc:492] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.832139 1183 ts_tablet_manager.cc:1403] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20251029 08:52:32.832899 2731 raft_consensus.cc:493] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.833065 2731 raft_consensus.cc:515] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.833158 1183 raft_consensus.cc:359] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.833334 1183 raft_consensus.cc:385] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.833392 1183 raft_consensus.cc:740] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.833767 2731 leader_election.cc:290] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.836473 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:32.836791 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:32.836966 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:32.837261 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 23dc7808f8da41e3aa0af5f318cd05c1 (table default.Table_b7290937457c4e9aad0d89f247f68a34 [id=677af5a976e14a88a896241f81fe2bc9]). Expected version 2 got 1
W20251029 08:52:32.837716 861 leader_election.cc:343] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:32.838351 899 tablet_service.cc:1505] Processing CreateTablet for tablet 3160e3945b294a099a1656956349ee5d (DEFAULT_TABLE table=default.Table_13bd1d85abb34cfa9e1f187cf0b8d566 [id=b0b50fe5316c494c8dedfef26fe185d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.838819 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3160e3945b294a099a1656956349ee5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.844103 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1222) took 2031 ms. Trace:
I20251029 08:52:32.844218 1176 rpcz_store.cc:276] 1029 08:52:30.812993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.824225 (+ 11232us) service_pool.cc:225] Handling call
1029 08:52:32.825695 (+2001470us) ts_tablet_manager.cc:1151] Deleting tablet e322478c9ba64c2a95a2c299123926dd
1029 08:52:32.843502 (+ 17807us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.844065 (+ 563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001293}
I20251029 08:52:32.847564 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 9f8348b6234e4dbe8bdb58edad34ea7b
I20251029 08:52:32.847862 1974 ts_tablet_manager.cc:1929] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:32.848060 588 catalog_manager.cc:6719] ReplaceTablet: tablet 9f8348b6234e4dbe8bdb58edad34ea7b deleted and replaced by tablet af576f0e9ccd4c88b8d3c7dd15df1785
W20251029 08:52:32.850870 861 leader_election.cc:343] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:32.851029 861 leader_election.cc:304] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:32.851266 2554 raft_consensus.cc:2749] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:32.861100 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1208) took 2198 ms. Trace:
I20251029 08:52:32.861232 1176 rpcz_store.cc:276] 1029 08:52:30.663018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.731292 (+ 68274us) service_pool.cc:225] Handling call
1029 08:52:32.844497 (+2113205us) ts_tablet_manager.cc:1151] Deleting tablet d87191d05d4145a796a8823f723bc499
1029 08:52:32.860058 (+ 15561us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.861074 (+ 1016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003748}
I20251029 08:52:32.847991 1974 log.cc:1199] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/e8c4d793f10147fba5e4f44474575bf0
I20251029 08:52:32.866634 1974 ts_tablet_manager.cc:1950] T e8c4d793f10147fba5e4f44474575bf0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:32.867717 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1061) took 6747 ms. Trace:
I20251029 08:52:32.867743 1176 tablet_replica.cc:333] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:32.868072 1176 raft_consensus.cc:2243] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:32.868214 1178 tablet_bootstrap.cc:492] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.868264 1176 raft_consensus.cc:2272] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.868374 1178 ts_tablet_manager.cc:1403] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.265s user 0.048s sys 0.110s
I20251029 08:52:32.868656 416 master-stress-test.cc:579] Killed test thread 21/25
I20251029 08:52:32.868762 416 master-stress-test.cc:577] Killing test thread 22/25
I20251029 08:52:32.868863 1176 ts_tablet_manager.cc:1916] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:32.868956 594 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: default.Table_b7290937457c4e9aad0d89f247f68a34 [id=677af5a976e14a88a896241f81fe2bc9] alter complete (version 2)
I20251029 08:52:32.869186 1974 rpcz_store.cc:276] 1029 08:52:26.120457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.121801 (+ 1344us) service_pool.cc:225] Handling call
1029 08:52:32.544905 (+6423104us) ts_tablet_manager.cc:1151] Deleting tablet e8c4d793f10147fba5e4f44474575bf0
1029 08:52:32.544916 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.805187 (+260271us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.847723 (+ 42536us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.867691 (+ 19968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6422836}
I20251029 08:52:32.869315 1178 raft_consensus.cc:359] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.869484 1178 raft_consensus.cc:385] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.869545 1178 raft_consensus.cc:740] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.869776 1178 consensus_queue.cc:260] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.870280 1178 ts_tablet_manager.cc:1434] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:32.871548 1178 tablet_bootstrap.cc:492] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:32.873333 1178 tablet_bootstrap.cc:654] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.874096 2938 ts_tablet_manager.cc:1929] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:32.874424 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 82589b43556c46559f0de4af7e8d9db0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82589b43556c46559f0de4af7e8d9db0 already in progress: creating tablet
W20251029 08:52:32.874680 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet ac09bb4502e440368633bbc90e9ad2cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac09bb4502e440368633bbc90e9ad2cf already in progress: creating tablet
I20251029 08:52:32.874925 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4200db822deb4ace936f561c230ac1fb (table default.Table_b6d28fc5fdc14b10b3436923d1b86bc1 [id=572a15f289894adb908547d8f011765a]) successfully deleted
I20251029 08:52:32.878366 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 00829c7fa0fe4e379a1a3044550e5941 (table default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]). Expected version 3 got 0
I20251029 08:52:32.879948 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet e8c4d793f10147fba5e4f44474575bf0 (table default.Table_02acde1d2bfe41e9a0f7565ed487c1c7 [id=6dc4e58a482742c19d68262a096d26c7]) successfully deleted
I20251029 08:52:32.880574 895 tablet_service.cc:1505] Processing CreateTablet for tablet e3d6cb6c7ad6455f87b8f21067d63687 (DEFAULT_TABLE table=default.Table_082ce20f2d2a450f9216c2c34a03f696 [id=886f566e4ea44c839c9c8954bb819214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.881116 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d6cb6c7ad6455f87b8f21067d63687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.884177 893 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c24c0d32f74435bd9d28f2415e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.885608 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2fb9ab76dcfe486e89bbe9391cab56f9 (table default.Table_86364f11715846199b42645e6deeb2fb [id=7e89e19d5a384f8195c22592fe17e43d]) successfully deleted
I20251029 08:52:32.890468 761 tablet_service.cc:1505] Processing CreateTablet for tablet 3160e3945b294a099a1656956349ee5d (DEFAULT_TABLE table=default.Table_13bd1d85abb34cfa9e1f187cf0b8d566 [id=b0b50fe5316c494c8dedfef26fe185d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.890973 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3160e3945b294a099a1656956349ee5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.901145 757 tablet_service.cc:1505] Processing CreateTablet for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc (DEFAULT_TABLE table=default.Table_2c16fc98c6a94cf7a42c8cb0413418d4 [id=b18017bf1d3e4d5fa654413525adeb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.901623 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.908648 901 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b with delete_type TABLET_DATA_DELETED (replaced by tablet af576f0e9ccd4c88b8d3c7dd15df1785) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:32.909941 1176 ts_tablet_manager.cc:1929] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:32.910068 1176 log.cc:1199] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a506e35afe6a4c87901f27c434f3b348
I20251029 08:52:32.910411 1176 ts_tablet_manager.cc:1950] T a506e35afe6a4c87901f27c434f3b348 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:32.911121 745 tablet_service.cc:1505] Processing CreateTablet for tablet e3d6cb6c7ad6455f87b8f21067d63687 (DEFAULT_TABLE table=default.Table_082ce20f2d2a450f9216c2c34a03f696 [id=886f566e4ea44c839c9c8954bb819214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.911463 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d6cb6c7ad6455f87b8f21067d63687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.915954 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b with delete_type TABLET_DATA_DELETED (replaced by tablet af576f0e9ccd4c88b8d3c7dd15df1785) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.916512 2938 log.cc:1199] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/0bd490da0b0e47fc965343c123a27b22
I20251029 08:52:32.916961 2938 ts_tablet_manager.cc:1950] T 0bd490da0b0e47fc965343c123a27b22 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:32.881548 1183 consensus_queue.cc:260] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.886600 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 3160e3945b294a099a1656956349ee5d (DEFAULT_TABLE table=default.Table_13bd1d85abb34cfa9e1f187cf0b8d566 [id=b0b50fe5316c494c8dedfef26fe185d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.918169 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3160e3945b294a099a1656956349ee5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.918218 1183 ts_tablet_manager.cc:1434] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.086s user 0.001s sys 0.000s
I20251029 08:52:32.918478 1183 tablet_bootstrap.cc:492] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.919986 1183 tablet_bootstrap.cc:654] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.921877 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1060) took 6801 ms. Trace:
I20251029 08:52:32.921998 1974 rpcz_store.cc:276] 1029 08:52:26.120348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.133632 (+ 13284us) service_pool.cc:225] Handling call
1029 08:52:32.876127 (+6742495us) ts_tablet_manager.cc:1151] Deleting tablet bbc4345cabbc476485b6a040a076ddaf
1029 08:52:32.876137 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.921850 (+ 45713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6742045}
I20251029 08:52:32.922384 1974 tablet_replica.cc:333] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:32.922609 1974 raft_consensus.cc:2243] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:32.922676 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d5bb0fd4c62d47a2b10719ebd55851c0 (table default.Table_11a78e5b37424289a184f8bccbd263d4 [id=c54d460a25104cc19f43a5517c203b31]) successfully deleted
I20251029 08:52:32.922727 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 533193918fdd4930b95806e5b2f62730 (DEFAULT_TABLE table=default.Table_cfb3215efbfa410a8c53f7adfcbe6d2a [id=acab9b7b2572497fa7c8fdcca90db965]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:32.922935 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f2c18e4c41714ba38bdd2e208a5e761e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c18e4c41714ba38bdd2e208a5e761e already in progress: creating tablet
I20251029 08:52:32.922974 1974 raft_consensus.cc:2272] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.923054 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533193918fdd4930b95806e5b2f62730. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.923121 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 3ee83f5526f74ba2a344edd97db94050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee83f5526f74ba2a344edd97db94050 already in progress: creating tablet
W20251029 08:52:32.923271 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e27701fbf5b44e90ba33bed99fdc4d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e27701fbf5b44e90ba33bed99fdc4d9b already in progress: creating tablet
W20251029 08:52:32.923432 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9e44e56554574d5abb67d08d88218152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e44e56554574d5abb67d08d88218152 already in progress: creating tablet
I20251029 08:52:32.923673 1974 ts_tablet_manager.cc:1916] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:32.923694 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0adba75cc7d74b1497de9af7ad7ccdd0 already in progress: creating tablet
I20251029 08:52:32.924938 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 31bed64a27164baa8177b87b54b5947b (table default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]) successfully deleted
W20251029 08:52:32.925124 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7492195a41664ef69554d8a6cb148e21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7492195a41664ef69554d8a6cb148e21 already in progress: creating tablet
W20251029 08:52:32.925259 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 4dec9fdf1ae94ae687f30f89096ce89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dec9fdf1ae94ae687f30f89096ce89b already in progress: creating tablet
W20251029 08:52:32.925379 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 07bf5d693be240f18bacccc81a6ea5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07bf5d693be240f18bacccc81a6ea5d4 already in progress: creating tablet
W20251029 08:52:32.925498 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc6c8f9ee9840c08c9436ab516a62d7 already in progress: creating tablet
W20251029 08:52:32.925612 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet e322478c9ba64c2a95a2c299123926dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e322478c9ba64c2a95a2c299123926dd already in progress: creating tablet
W20251029 08:52:32.925729 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d87191d05d4145a796a8823f723bc499 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d87191d05d4145a796a8823f723bc499 already in progress: creating tablet
I20251029 08:52:32.927178 757 tablet_service.cc:1505] Processing CreateTablet for tablet 533193918fdd4930b95806e5b2f62730 (DEFAULT_TABLE table=default.Table_cfb3215efbfa410a8c53f7adfcbe6d2a [id=acab9b7b2572497fa7c8fdcca90db965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.886984 1050 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.927613 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533193918fdd4930b95806e5b2f62730. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.929316 1178 tablet_bootstrap.cc:492] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:32.888641 1044 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3a2fab467348099a4a1fec74eb5e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.929476 1178 ts_tablet_manager.cc:1403] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.025s
I20251029 08:52:32.888919 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a9fe3e9a507441b8ba5953c0120289c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.889182 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 85df43bf781c421688f163331de0eb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:18 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.889650 1043 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f8348b6234e4dbe8bdb58edad34ea7b with delete_type TABLET_DATA_DELETED (replaced by tablet af576f0e9ccd4c88b8d3c7dd15df1785) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:32.890727 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc (DEFAULT_TABLE table=default.Table_2c16fc98c6a94cf7a42c8cb0413418d4 [id=b18017bf1d3e4d5fa654413525adeb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.929983 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.930394 1178 raft_consensus.cc:359] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.930625 1178 raft_consensus.cc:385] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.930685 1178 raft_consensus.cc:740] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.930933 1178 consensus_queue.cc:260] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.931521 1178 ts_tablet_manager.cc:1434] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:32.910805 1049 tablet_service.cc:1505] Processing CreateTablet for tablet e3d6cb6c7ad6455f87b8f21067d63687 (DEFAULT_TABLE table=default.Table_082ce20f2d2a450f9216c2c34a03f696 [id=886f566e4ea44c839c9c8954bb819214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.932283 1049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d6cb6c7ad6455f87b8f21067d63687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.936048 1183 tablet_bootstrap.cc:492] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:32.936174 1183 ts_tablet_manager.cc:1403] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251029 08:52:32.936828 1183 raft_consensus.cc:359] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.937067 1183 raft_consensus.cc:385] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:32.937115 1183 raft_consensus.cc:740] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:32.937294 1183 consensus_queue.cc:260] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.937721 1183 ts_tablet_manager.cc:1434] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 08:52:32.937983 1183 tablet_bootstrap.cc:492] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:32.940243 1183 tablet_bootstrap.cc:654] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:32.948158 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4518323963884288881118983b2aacd6 (DEFAULT_TABLE table=default.Table_750150daf42a46f0a5f4f2a4d8925dd5 [id=860218ecd80d4eeba8455dee8b543a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.948535 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4518323963884288881118983b2aacd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.948689 759 tablet_service.cc:1505] Processing CreateTablet for tablet 475608edbca040a0b7e492a36b4f1184 (DEFAULT_TABLE table=default.Table_33ff6ee160ac45db94ce4a84c3d5ad61 [id=afe8b22ad3514bd595328af278d1340f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.949116 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475608edbca040a0b7e492a36b4f1184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.958209 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 0bd490da0b0e47fc965343c123a27b22 (table default.Table_a97711da23ff4ab79b7f1810e23031c6 [id=0d9f935637a8420b985538e9de4e21ca]) successfully deleted
I20251029 08:52:32.941006 1043 tablet_service.cc:1505] Processing CreateTablet for tablet e9a8dda9dd274207aa8ec87e691855a2 (DEFAULT_TABLE table=default.Table_3ee1d43136654538bff6cb34c23ff8da [id=4611bb0c520b450eb6f3ff2096f1164e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.959157 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a8dda9dd274207aa8ec87e691855a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.961922 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet a17142ffce3e4c359e715ef63747c40d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17142ffce3e4c359e715ef63747c40d already in progress: creating tablet
I20251029 08:52:32.962076 752 tablet_service.cc:1505] Processing CreateTablet for tablet e9a8dda9dd274207aa8ec87e691855a2 (DEFAULT_TABLE table=default.Table_3ee1d43136654538bff6cb34c23ff8da [id=4611bb0c520b450eb6f3ff2096f1164e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.962651 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a8dda9dd274207aa8ec87e691855a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:32.962725 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet d010353b251140e99ce2df996f965991 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d010353b251140e99ce2df996f965991 already in progress: creating tablet
W20251029 08:52:32.962913 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 1b3ab5946c574171b66f206c85720f79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b3ab5946c574171b66f206c85720f79 already in progress: creating tablet
W20251029 08:52:32.963080 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet f8a896408fd142b9b41d606738234ba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a896408fd142b9b41d606738234ba4 already in progress: creating tablet
I20251029 08:52:32.963709 758 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fad7549758649229ad771de926aaf15 with delete_type TABLET_DATA_DELETED (replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:32.966189 2896 raft_consensus.cc:493] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:32.966334 2896 raft_consensus.cc:515] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.947873 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 4518323963884288881118983b2aacd6 (DEFAULT_TABLE table=default.Table_750150daf42a46f0a5f4f2a4d8925dd5 [id=860218ecd80d4eeba8455dee8b543a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.966531 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:32.967005 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4518323963884288881118983b2aacd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.967146 2896 leader_election.cc:290] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.967712 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:32.967885 774 raft_consensus.cc:2468] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:32.968607 998 leader_election.cc:304] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:32.968919 2896 raft_consensus.cc:2804] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:32.968990 2896 raft_consensus.cc:493] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:32.969058 2896 raft_consensus.cc:3060] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.977826 2896 raft_consensus.cc:515] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.954531 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 475608edbca040a0b7e492a36b4f1184 (DEFAULT_TABLE table=default.Table_33ff6ee160ac45db94ce4a84c3d5ad61 [id=afe8b22ad3514bd595328af278d1340f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.979689 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475608edbca040a0b7e492a36b4f1184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.981688 2920 consensus_queue.cc:1048] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:32.981887 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:32.982072 774 raft_consensus.cc:3060] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:32.984858 774 raft_consensus.cc:2468] T 471737aa35e94dcfaa2b3451cc8f2014 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:32.985317 998 leader_election.cc:304] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:32.985570 2359 raft_consensus.cc:2804] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:32.985663 2359 raft_consensus.cc:697] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:32.985894 2359 consensus_queue.cc:237] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:32.987262 2896 leader_election.cc:290] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:32.957839 1047 tablet_service.cc:1505] Processing CreateTablet for tablet e2d7a497b65c4671bf4730c5cf97f222 (DEFAULT_TABLE table=default.Table_bc2ad33f108841bf8d203c9182d05644 [id=8781be03a77748ff9b8565905470df1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:32.988595 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d7a497b65c4671bf4730c5cf97f222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:32.994668 2938 tablet_replica.cc:333] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:32.994918 2938 raft_consensus.cc:2243] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:32.995226 2938 raft_consensus.cc:2272] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:32.995860 2938 ts_tablet_manager.cc:1916] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.003535 1183 tablet_bootstrap.cc:492] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.006052 1183 ts_tablet_manager.cc:1403] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20251029 08:52:33.007082 1183 raft_consensus.cc:359] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:33.009891 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet cb1e2c24e30048abb3ae9fb92171e049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb1e2c24e30048abb3ae9fb92171e049 already in progress: creating tablet
W20251029 08:52:33.010327 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet c7818ad8e5ec4fca8e3b25019f545dd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7818ad8e5ec4fca8e3b25019f545dd8 already in progress: creating tablet
I20251029 08:52:33.011444 758 tablet_service.cc:1505] Processing CreateTablet for tablet e2d7a497b65c4671bf4730c5cf97f222 (DEFAULT_TABLE table=default.Table_bc2ad33f108841bf8d203c9182d05644 [id=8781be03a77748ff9b8565905470df1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.011858 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d7a497b65c4671bf4730c5cf97f222. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:33.017428 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bbc4345cabbc476485b6a040a076ddaf because the tablet was not found. No further retry: Not found: Tablet not found: bbc4345cabbc476485b6a040a076ddaf
I20251029 08:52:33.022719 1183 raft_consensus.cc:385] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.023130 1183 raft_consensus.cc:740] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.023981 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 31382455aea44849a60a3da772fbc9b8 (DEFAULT_TABLE table=default.Table_d1f52fb5254c46f399a8df1a241755a5 [id=e6254beeb0204a35ba9647435da2e7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.024449 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31382455aea44849a60a3da772fbc9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.024940 764 tablet_service.cc:1505] Processing CreateTablet for tablet 31382455aea44849a60a3da772fbc9b8 (DEFAULT_TABLE table=default.Table_d1f52fb5254c46f399a8df1a241755a5 [id=e6254beeb0204a35ba9647435da2e7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.025286 764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31382455aea44849a60a3da772fbc9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.025434 901 tablet_service.cc:1505] Processing CreateTablet for tablet 533193918fdd4930b95806e5b2f62730 (DEFAULT_TABLE table=default.Table_cfb3215efbfa410a8c53f7adfcbe6d2a [id=acab9b7b2572497fa7c8fdcca90db965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.026242 893 tablet_service.cc:1505] Processing CreateTablet for tablet e9a8dda9dd274207aa8ec87e691855a2 (DEFAULT_TABLE table=default.Table_3ee1d43136654538bff6cb34c23ff8da [id=4611bb0c520b450eb6f3ff2096f1164e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.026659 892 tablet_service.cc:1505] Processing CreateTablet for tablet 475608edbca040a0b7e492a36b4f1184 (DEFAULT_TABLE table=default.Table_33ff6ee160ac45db94ce4a84c3d5ad61 [id=afe8b22ad3514bd595328af278d1340f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.027068 898 tablet_service.cc:1505] Processing CreateTablet for tablet 4518323963884288881118983b2aacd6 (DEFAULT_TABLE table=default.Table_750150daf42a46f0a5f4f2a4d8925dd5 [id=860218ecd80d4eeba8455dee8b543a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.027431 890 tablet_service.cc:1505] Processing CreateTablet for tablet e2d7a497b65c4671bf4730c5cf97f222 (DEFAULT_TABLE table=default.Table_bc2ad33f108841bf8d203c9182d05644 [id=8781be03a77748ff9b8565905470df1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.027805 891 tablet_service.cc:1505] Processing CreateTablet for tablet 31382455aea44849a60a3da772fbc9b8 (DEFAULT_TABLE table=default.Table_d1f52fb5254c46f399a8df1a241755a5 [id=e6254beeb0204a35ba9647435da2e7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.028203 900 tablet_service.cc:1505] Processing CreateTablet for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.028575 896 tablet_service.cc:1505] Processing CreateTablet for tablet 968445d6d58a4dd5b81a67b6ff41d4a4 (DEFAULT_TABLE table=default.Table_7ec9d338fa86438da982cb2b33730cd4 [id=30e0be20ea2348b6bd55cf91b2973357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.028970 903 tablet_service.cc:1505] Processing CreateTablet for tablet 5df60af38a7f4ee5a580f1b6cfab020e (DEFAULT_TABLE table=default.Table_6f7db122e41f478290349f3c2a281ac9 [id=4647c8724038469597f9c135e8f2e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.029795 905 tablet_service.cc:1505] Processing CreateTablet for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc (DEFAULT_TABLE table=default.Table_2c16fc98c6a94cf7a42c8cb0413418d4 [id=b18017bf1d3e4d5fa654413525adeb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.032224 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.038421 751 tablet_service.cc:1505] Processing CreateTablet for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.038856 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.023972 1183 consensus_queue.cc:260] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.046922 1183 ts_tablet_manager.cc:1434] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251029 08:52:33.047837 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1225) took 2185 ms. Trace:
I20251029 08:52:33.047940 1176 rpcz_store.cc:276] 1029 08:52:30.862797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.863176 (+ 379us) service_pool.cc:225] Handling call
1029 08:52:32.867661 (+2004485us) ts_tablet_manager.cc:1151] Deleting tablet a506e35afe6a4c87901f27c434f3b348
1029 08:52:32.867677 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.906756 (+ 39079us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:32.909862 (+ 3106us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.047810 (+137948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004232}
I20251029 08:52:33.048514 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1230) took 2063 ms. Trace:
I20251029 08:52:33.048976 755 tablet_service.cc:1505] Processing CreateTablet for tablet 968445d6d58a4dd5b81a67b6ff41d4a4 (DEFAULT_TABLE table=default.Table_7ec9d338fa86438da982cb2b33730cd4 [id=30e0be20ea2348b6bd55cf91b2973357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.049458 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968445d6d58a4dd5b81a67b6ff41d4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.049456 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a506e35afe6a4c87901f27c434f3b348 (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281]) successfully deleted
I20251029 08:52:33.049986 1183 tablet_bootstrap.cc:492] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.048840 1047 tablet_service.cc:1505] Processing CreateTablet for tablet 968445d6d58a4dd5b81a67b6ff41d4a4 (DEFAULT_TABLE table=default.Table_7ec9d338fa86438da982cb2b33730cd4 [id=30e0be20ea2348b6bd55cf91b2973357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.050959 1047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968445d6d58a4dd5b81a67b6ff41d4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.049077 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet d26ff78aa5f94ae49a197daf2be00aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.049448 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 5df60af38a7f4ee5a580f1b6cfab020e (DEFAULT_TABLE table=default.Table_6f7db122e41f478290349f3c2a281ac9 [id=4647c8724038469597f9c135e8f2e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.051534 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df60af38a7f4ee5a580f1b6cfab020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.052506 1178 tablet_bootstrap.cc:492] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.052441 894 tablet_service.cc:1505] Processing CreateTablet for tablet a2d3cdad64084fc78d929279c211c855 (DEFAULT_TABLE table=default.Table_10da20471c1a49ec93301ed22cf79f09 [id=f32177561252405aa8bd0f89201ab9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.054816 1178 tablet_bootstrap.cc:654] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.059777 757 tablet_service.cc:1505] Processing CreateTablet for tablet 5df60af38a7f4ee5a580f1b6cfab020e (DEFAULT_TABLE table=default.Table_6f7db122e41f478290349f3c2a281ac9 [id=4647c8724038469597f9c135e8f2e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.059965 2920 raft_consensus.cc:493] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.060261 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df60af38a7f4ee5a580f1b6cfab020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.060173 2920 raft_consensus.cc:515] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.060417 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d7a497b65c4671bf4730c5cf97f222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.060636 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31382455aea44849a60a3da772fbc9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.060803 905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcfe827d13e49c9b4d5d8b4dce67fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.060846 2920 leader_election.cc:290] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.061224 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df60af38a7f4ee5a580f1b6cfab020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.061394 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475608edbca040a0b7e492a36b4f1184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.061530 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968445d6d58a4dd5b81a67b6ff41d4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.061729 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a8dda9dd274207aa8ec87e691855a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.061952 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533193918fdd4930b95806e5b2f62730. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.063524 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4518323963884288881118983b2aacd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.064278 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f36641b7086430a89380056e04b8aaf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.064896 718 leader_election.cc:343] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.065264 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f36641b7086430a89380056e04b8aaf" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:33.065891 720 leader_election.cc:343] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.066025 720 leader_election.cc:304] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.066506 2920 raft_consensus.cc:2749] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.067066 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.067515 1043 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9fce57b2f541ed994dc119d91b47b1 (DEFAULT_TABLE table=default.Table_e0e268e3dcf941c18ac578d3de6c40c1 [id=72d145753df44690b9717ed13708cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.067906 1043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9fce57b2f541ed994dc119d91b47b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.068202 1974 ts_tablet_manager.cc:1929] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.068361 1974 log.cc:1199] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/7be4bcf9e3324bd884e946fae20648fd
I20251029 08:52:33.068733 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d (DEFAULT_TABLE table=default.Table_680fa9c86aeb41c7aaa9bc8937675362 [id=3436cd1f0dc84d06af2b3fea03a94c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.069136 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a2ae3e3bdd4ae58e4dbdbf2f42807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.072870 759 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9fce57b2f541ed994dc119d91b47b1 (DEFAULT_TABLE table=default.Table_e0e268e3dcf941c18ac578d3de6c40c1 [id=72d145753df44690b9717ed13708cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.073310 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9fce57b2f541ed994dc119d91b47b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.073598 1183 tablet_bootstrap.cc:654] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.074299 1178 tablet_bootstrap.cc:492] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.074425 1178 ts_tablet_manager.cc:1403] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20251029 08:52:33.074992 1038 tablet_service.cc:1505] Processing CreateTablet for tablet a2d3cdad64084fc78d929279c211c855 (DEFAULT_TABLE table=default.Table_10da20471c1a49ec93301ed22cf79f09 [id=f32177561252405aa8bd0f89201ab9e8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 08:52:33.075165 902 consensus_meta.cc:287] T 2ce6cacedfdf4cf8b71dba57ad3036ac P 927689e0dc5e4bc79d4a9d79a873f088: Time spent flushing consensus metadata: real 0.805s user 0.025s sys 0.000s
I20251029 08:52:33.075196 1178 raft_consensus.cc:359] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.075376 1178 raft_consensus.cc:385] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.075423 1178 raft_consensus.cc:740] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.075618 1178 consensus_queue.cc:260] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.075829 750 tablet_service.cc:1505] Processing CreateTablet for tablet a2d3cdad64084fc78d929279c211c855 (DEFAULT_TABLE table=default.Table_10da20471c1a49ec93301ed22cf79f09 [id=f32177561252405aa8bd0f89201ab9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.076188 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3cdad64084fc78d929279c211c855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.048609 1176 rpcz_store.cc:276] 1029 08:52:30.984886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.984955 (+ 69us) service_pool.cc:225] Handling call
1029 08:52:33.048369 (+2063414us) ts_tablet_manager.cc:1151] Deleting tablet 0b0f1cdfb1804d1c8d6220907795fc80
1029 08:52:33.048382 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.048496 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063225}
I20251029 08:52:33.076795 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1238) took 1833 ms. Trace:
I20251029 08:52:33.076882 1176 rpcz_store.cc:276] 1029 08:52:31.243473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.243537 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:33.076612 (+1833075us) ts_tablet_manager.cc:1151] Deleting tablet 5ad98112c0ea4f59980e72165cef8a1f
1029 08:52:33.076620 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.076777 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832887}
I20251029 08:52:33.078584 1176 tablet_replica.cc:333] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.078783 1176 raft_consensus.cc:2243] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.078926 1176 raft_consensus.cc:2272] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.079453 1176 ts_tablet_manager.cc:1916] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:33.085232 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 0b0f1cdfb1804d1c8d6220907795fc80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b0f1cdfb1804d1c8d6220907795fc80 already in progress: creating tablet
W20251029 08:52:33.085512 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 5ad98112c0ea4f59980e72165cef8a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ad98112c0ea4f59980e72165cef8a1f already in progress: creating tablet
I20251029 08:52:33.086424 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 527a83465a844849811b0f9d26428af0 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]). Expected version 1 got 0
I20251029 08:52:33.076201 1178 ts_tablet_manager.cc:1434] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:33.093783 2938 ts_tablet_manager.cc:1929] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:33.094151 1178 tablet_bootstrap.cc:492] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.102880 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa]). Expected version 2 got 0
I20251029 08:52:33.103850 1178 tablet_bootstrap.cc:654] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.104085 2938 log.cc:1199] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2fafe9a0731b48b3a60032d94623681d
I20251029 08:52:33.104350 595 catalog_manager.cc:5649] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:33.104599 2938 ts_tablet_manager.cc:1950] T 2fafe9a0731b48b3a60032d94623681d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.105346 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:33.106400 1176 ts_tablet_manager.cc:1929] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.106544 1176 log.cc:1199] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/62597189ecb047788b83788b514e9099
I20251029 08:52:33.106863 1176 ts_tablet_manager.cc:1950] T 62597189ecb047788b83788b514e9099 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.075505 902 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50036 (request call id 1316) took 1314 ms. Trace:
I20251029 08:52:33.116339 902 rpcz_store.cc:276] 1029 08:52:31.761457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:32.232581 (+471124us) service_pool.cc:225] Handling call
1029 08:52:32.233485 (+ 904us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 08:52:32.233553 (+ 68us) ts_tablet_manager.cc:682] Creating new metadata...
1029 08:52:32.270383 (+ 36830us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.075315 (+804932us) tablet_replica.cc:206] Creating consensus instance
1029 08:52:33.075486 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 08:52:33.118772 887 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9fce57b2f541ed994dc119d91b47b1 (DEFAULT_TABLE table=default.Table_e0e268e3dcf941c18ac578d3de6c40c1 [id=72d145753df44690b9717ed13708cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.119359 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9fce57b2f541ed994dc119d91b47b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 08:52:33.126529 996 leader_election.cc:343] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.128723 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3cdad64084fc78d929279c211c855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.132571 1974 ts_tablet_manager.cc:1950] T 7be4bcf9e3324bd884e946fae20648fd P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.139863 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "471737aa35e94dcfaa2b3451cc8f2014" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:33.148011 1178 tablet_bootstrap.cc:492] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.148252 1178 ts_tablet_manager.cc:1403] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.009s
I20251029 08:52:33.149372 1178 raft_consensus.cc:359] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.149609 1178 raft_consensus.cc:385] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.149670 1178 raft_consensus.cc:740] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.149968 1178 consensus_queue.cc:260] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.150575 1178 ts_tablet_manager.cc:1434] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:33.163400 1178 tablet_bootstrap.cc:492] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.163697 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1239) took 1917 ms. Trace:
I20251029 08:52:33.163810 1176 rpcz_store.cc:276] 1029 08:52:31.245981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.246030 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:33.078076 (+1832046us) ts_tablet_manager.cc:1151] Deleting tablet 62597189ecb047788b83788b514e9099
1029 08:52:33.078548 (+ 472us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.102992 (+ 24444us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.106298 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.163669 (+ 57371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1831882}
I20251029 08:52:33.164428 1176 tablet_replica.cc:333] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.164732 1176 raft_consensus.cc:2243] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.164927 1176 raft_consensus.cc:2272] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.165256 1178 tablet_bootstrap.cc:654] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.165612 1176 ts_tablet_manager.cc:1916] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.173004 1176 ts_tablet_manager.cc:1929] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.173148 1176 log.cc:1199] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a9fe3e9a507441b8ba5953c0120289c3
I20251029 08:52:33.173441 1176 ts_tablet_manager.cc:1950] T a9fe3e9a507441b8ba5953c0120289c3 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
W20251029 08:52:33.175814 861 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 141114us
I20251029 08:52:33.179456 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1240) took 1914 ms. Trace:
I20251029 08:52:33.179581 1176 rpcz_store.cc:276] 1029 08:52:31.264995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.265051 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:33.164129 (+1899078us) ts_tablet_manager.cc:1151] Deleting tablet a9fe3e9a507441b8ba5953c0120289c3
1029 08:52:33.164359 (+ 230us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.169562 (+ 5203us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.172888 (+ 3326us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.179421 (+ 6533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898893}
I20251029 08:52:33.182794 1178 tablet_bootstrap.cc:492] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.182991 1178 ts_tablet_manager.cc:1403] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20251029 08:52:33.183153 1176 tablet_replica.cc:333] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.183439 1176 raft_consensus.cc:2243] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.183665 1176 raft_consensus.cc:2272] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.184322 1176 ts_tablet_manager.cc:1916] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.189656 1177 tablet_bootstrap.cc:492] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.189872 1177 ts_tablet_manager.cc:1403] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.460s user 0.032s sys 0.065s
I20251029 08:52:33.191733 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1059) took 7071 ms. Trace:
I20251029 08:52:33.191840 1974 rpcz_store.cc:276] 1029 08:52:26.120206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.168597 (+ 48391us) service_pool.cc:225] Handling call
1029 08:52:32.922315 (+6753718us) ts_tablet_manager.cc:1151] Deleting tablet 7be4bcf9e3324bd884e946fae20648fd
1029 08:52:32.922330 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:32.945127 (+ 22797us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.068108 (+122981us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.191710 (+123602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6747869}
I20251029 08:52:33.192301 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1075) took 7005 ms. Trace:
I20251029 08:52:33.192373 1974 rpcz_store.cc:276] 1029 08:52:26.187030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.188387 (+ 1357us) service_pool.cc:225] Handling call
1029 08:52:33.192155 (+7003768us) ts_tablet_manager.cc:1151] Deleting tablet 919e5494b84343639a2b9b81b05f1c2c
1029 08:52:33.192168 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.192282 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7003189}
I20251029 08:52:33.194093 1178 raft_consensus.cc:359] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.194403 1178 raft_consensus.cc:385] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.194482 1178 raft_consensus.cc:740] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.194564 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1077) took 6993 ms. Trace:
I20251029 08:52:33.194648 1974 rpcz_store.cc:276] 1029 08:52:26.201410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.201490 (+ 80us) service_pool.cc:225] Handling call
1029 08:52:33.192578 (+6991088us) ts_tablet_manager.cc:1151] Deleting tablet 3c9cdcec8e494050a475defd0cfd119f
1029 08:52:33.192586 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.194543 (+ 1957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6990881}
I20251029 08:52:33.194736 1178 consensus_queue.cc:260] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.194972 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1079) took 6973 ms. Trace:
I20251029 08:52:33.195047 1974 rpcz_store.cc:276] 1029 08:52:26.221981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.222035 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:33.194877 (+6972842us) ts_tablet_manager.cc:1151] Deleting tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac
1029 08:52:33.194885 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.194957 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6972660}
I20251029 08:52:33.195295 1974 tablet_replica.cc:333] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.195384 1178 ts_tablet_manager.cc:1434] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 08:52:33.195431 884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 190) took 1413 ms. Trace:
I20251029 08:52:33.195497 884 rpcz_store.cc:276] 1029 08:52:31.782081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.784077 (+ 1996us) service_pool.cc:225] Handling call
1029 08:52:33.195416 (+1411339us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 08:52:33.195745 1178 tablet_bootstrap.cc:492] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.198555 1178 tablet_bootstrap.cc:654] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.200256 881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 191) took 1413 ms. Trace:
I20251029 08:52:33.200362 881 rpcz_store.cc:276] 1029 08:52:31.786798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.786864 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:33.200205 (+1413341us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251029 08:52:33.210198 996 leader_election.cc:343] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.217726 1177 raft_consensus.cc:359] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.218685 1177 raft_consensus.cc:385] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.218930 1177 raft_consensus.cc:740] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.219261 1177 consensus_queue.cc:260] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.223543 1177 ts_tablet_manager.cc:1434] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20251029 08:52:33.224300 1177 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.235793 2964 raft_consensus.cc:493] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.236169 2964 raft_consensus.cc:515] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.237069 2964 leader_election.cc:290] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:33.237547 1176 ts_tablet_manager.cc:1929] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.237654 1176 log.cc:1199] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/34729c3f97a44713a352bde297a80027
I20251029 08:52:33.238009 1176 ts_tablet_manager.cc:1950] T 34729c3f97a44713a352bde297a80027 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.238734 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1241) took 1959 ms. Trace:
I20251029 08:52:33.238832 1176 rpcz_store.cc:276] 1029 08:52:31.279297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.279353 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:33.183078 (+1903725us) ts_tablet_manager.cc:1151] Deleting tablet 34729c3f97a44713a352bde297a80027
1029 08:52:33.183093 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.191737 (+ 8644us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.237422 (+ 45685us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.238717 (+ 1295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900552}
I20251029 08:52:33.243901 1176 tablet_replica.cc:333] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.244131 1176 raft_consensus.cc:2243] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.244288 1176 raft_consensus.cc:2272] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.244815 1176 ts_tablet_manager.cc:1916] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.111742 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 527a83465a844849811b0f9d26428af0 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]). Expected version 1 got 0
I20251029 08:52:33.166916 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 62597189ecb047788b83788b514e9099 (table default.Table_6056be95de0a4b34ab9dcf62b2968444 [id=c14f386ee208443db44d014c6b613070]) successfully deleted
W20251029 08:52:33.255162 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40969, user_credentials={real_user=slave}} blocked reactor thread for 88250us
I20251029 08:52:33.255721 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a9fe3e9a507441b8ba5953c0120289c3 (table default.Table_d74312a6b49e490ab01040a51e1e75e7 [id=d45481f7479f46d3bc3b014c7a437f55]) successfully deleted
W20251029 08:52:33.193814 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f7a6ef03f18c4ee78f2f10f31d1d8058 (table default.Table_e67f5ce3a1ac4caab580b9e7727e6d4a [id=afc30e0fa2464e558fcc3a7961a470e9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f7a6ef03f18c4ee78f2f10f31d1d8058 consensus state has no leader
I20251029 08:52:33.256336 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 34729c3f97a44713a352bde297a80027 (table default.Table_15cd2138e80e429db3004528a89ec2bf [id=a8a2e644aa73491b9d22e81babf2e582]) successfully deleted
I20251029 08:52:33.226433 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 7be4bcf9e3324bd884e946fae20648fd (table default.Table_3d5e7e9f16c04071af70d3a47624570d [id=0b1d4aab11c3418ca99dbbf604311dae]) successfully deleted
W20251029 08:52:33.256668 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 919e5494b84343639a2b9b81b05f1c2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 919e5494b84343639a2b9b81b05f1c2c already in progress: creating tablet
W20251029 08:52:33.257135 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3c9cdcec8e494050a475defd0cfd119f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c9cdcec8e494050a475defd0cfd119f already in progress: creating tablet
W20251029 08:52:33.257288 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61e7a2cbe0ff43b193f8c9ec7f3159ac already in progress: creating tablet
I20251029 08:52:33.265790 594 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 85a806cc04e641d9a1c4af97a24b13de (table default.Table_02acde1d2bfe41e9a0f7565ed487c1c7 [id=6dc4e58a482742c19d68262a096d26c7]). Expected version 1 got 0
I20251029 08:52:33.270166 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9c29b93a643ec837169a20c59de33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.273185 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "527a83465a844849811b0f9d26428af0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.273777 1176 ts_tablet_manager.cc:1929] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.273903 1176 log.cc:1199] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/58932f38c0d64aec9a02238eaea56a25
W20251029 08:52:33.273940 718 leader_election.cc:343] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.274219 1176 ts_tablet_manager.cc:1950] T 58932f38c0d64aec9a02238eaea56a25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.274967 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1242) took 1954 ms. Trace:
I20251029 08:52:33.275061 1176 rpcz_store.cc:276] 1029 08:52:31.320133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.320188 (+ 55us) service_pool.cc:225] Handling call
1029 08:52:33.243845 (+1923657us) ts_tablet_manager.cc:1151] Deleting tablet 58932f38c0d64aec9a02238eaea56a25
1029 08:52:33.243857 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.260229 (+ 16372us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.273095 (+ 12866us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.274949 (+ 1854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923485}
I20251029 08:52:33.275444 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 58932f38c0d64aec9a02238eaea56a25 (table default.Table_57929518185c4558a0f9f12c40945be2 [id=60ed38dd0d6e4bc5851de7c04b92abfc]) successfully deleted
I20251029 08:52:33.277411 1176 tablet_replica.cc:333] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.277619 1176 raft_consensus.cc:2243] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.277798 1176 raft_consensus.cc:2272] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.278326 1176 ts_tablet_manager.cc:1916] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.280094 1066 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "527a83465a844849811b0f9d26428af0" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:33.280676 720 leader_election.cc:343] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.280788 720 leader_election.cc:304] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.281374 2964 raft_consensus.cc:2749] T 527a83465a844849811b0f9d26428af0 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.281857 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3cdad64084fc78d929279c211c855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.286571 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]) successfully deleted
I20251029 08:52:33.286283 2938 tablet_replica.cc:333] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.289503 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fafe9a0731b48b3a60032d94623681d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:32 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.293290 2938 raft_consensus.cc:2243] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.293489 2938 raft_consensus.cc:2272] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.294225 2938 ts_tablet_manager.cc:1916] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.300247 1176 ts_tablet_manager.cc:1929] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:33.300360 1176 log.cc:1199] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/e1e2be77182e4259b76d20cae4e9e478
I20251029 08:52:33.300699 1176 ts_tablet_manager.cc:1950] T e1e2be77182e4259b76d20cae4e9e478 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.301414 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1243) took 1948 ms. Trace:
I20251029 08:52:33.301513 1176 rpcz_store.cc:276] 1029 08:52:31.353458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.353810 (+ 352us) service_pool.cc:225] Handling call
1029 08:52:33.277360 (+1923550us) ts_tablet_manager.cc:1151] Deleting tablet e1e2be77182e4259b76d20cae4e9e478
1029 08:52:33.277370 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.281546 (+ 4176us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.300139 (+ 18593us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.301395 (+ 1256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923353}
I20251029 08:52:33.302948 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 62597189ecb047788b83788b514e9099 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:33.310837 718 consensus_peers.cc:597] T 6c4e208427b94bd79bcb5c1080647731 P 7a8d80098fcb40ada903c3b2bacc2ea0 -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:33.311604 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be4bcf9e3324bd884e946fae20648fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.313190 1974 raft_consensus.cc:2243] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.314637 1974 raft_consensus.cc:2272] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.315189 1974 ts_tablet_manager.cc:1916] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.325641 1974 ts_tablet_manager.cc:1929] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.326092 1974 log.cc:1199] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/9adbcf3f12104fb5bfd1d9dd5c946c54
I20251029 08:52:33.327025 1974 ts_tablet_manager.cc:1950] T 9adbcf3f12104fb5bfd1d9dd5c946c54 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.328006 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1080) took 7096 ms. Trace:
I20251029 08:52:33.328521 1974 rpcz_store.cc:276] 1029 08:52:26.231951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.232011 (+ 60us) service_pool.cc:225] Handling call
1029 08:52:33.195259 (+6963248us) ts_tablet_manager.cc:1151] Deleting tablet 9adbcf3f12104fb5bfd1d9dd5c946c54
1029 08:52:33.195266 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.318316 (+123050us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.325542 (+ 7226us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.327991 (+ 2449us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117707,"tablet-delete.queue_time_us":6963045}
I20251029 08:52:33.329278 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1081) took 7092 ms. Trace:
I20251029 08:52:33.329694 1974 rpcz_store.cc:276] 1029 08:52:26.236591 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.236650 (+ 59us) service_pool.cc:225] Handling call
1029 08:52:33.329157 (+7092507us) ts_tablet_manager.cc:1151] Deleting tablet 59ad04f57c29407b9d2420520c095a28
1029 08:52:33.329167 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.329261 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7092329}
I20251029 08:52:33.330513 1974 tablet_replica.cc:333] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.330804 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 9adbcf3f12104fb5bfd1d9dd5c946c54 (table default.Table_2c072545dc7446eabcdbe80f548c5df7 [id=ed388d60c4864cd1bacb16c26cd7b07f]) successfully deleted
I20251029 08:52:33.331328 1974 raft_consensus.cc:2243] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:33.331642 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 59ad04f57c29407b9d2420520c095a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ad04f57c29407b9d2420520c095a28 already in progress: creating tablet
I20251029 08:52:33.332389 1974 raft_consensus.cc:2272] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.333691 1974 ts_tablet_manager.cc:1916] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.347503 2938 ts_tablet_manager.cc:1929] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.347826 2938 log.cc:1199] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d444a5d489434696ae178c6d650bf8d5
I20251029 08:52:33.347950 2896 raft_consensus.cc:493] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.348223 2938 ts_tablet_manager.cc:1950] T d444a5d489434696ae178c6d650bf8d5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.348192 2896 raft_consensus.cc:515] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.349305 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d444a5d489434696ae178c6d650bf8d5 (table default.Table_55491b160d0e4e6185be2e0fee2e67a5 [id=8431eac1a49a41c080964575e16345d7]) successfully deleted
I20251029 08:52:33.349303 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:33.349622 2938 tablet_replica.cc:333] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.349849 2938 raft_consensus.cc:2243] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:33.349938 998 leader_election.cc:343] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: d26ff78aa5f94ae49a197daf2be00aa5
I20251029 08:52:33.350227 2938 raft_consensus.cc:2272] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.350543 2896 leader_election.cc:290] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.351027 2938 ts_tablet_manager.cc:1916] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.353600 1974 ts_tablet_manager.cc:1929] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 08:52:33.357939 2938 ts_tablet_manager.cc:1929] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.360389 1974 log.cc:1199] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c9ed0a270b794e469d2fdbd0ea1d96f8
I20251029 08:52:33.361018 1974 ts_tablet_manager.cc:1950] T c9ed0a270b794e469d2fdbd0ea1d96f8 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.360985 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6c4e208427b94bd79bcb5c1080647731 (table default.Table_8df249abd861403db422e95addc8252c [id=064c00240f9c48ceac41175a2b8306bb]). Expected version 1 got 0
I20251029 08:52:33.361105 2938 log.cc:1199] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b8f2b34642704cd6bb966b6dcd494d51
I20251029 08:52:33.361665 2938 ts_tablet_manager.cc:1950] T b8f2b34642704cd6bb966b6dcd494d51 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.362020 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1083) took 7105 ms. Trace:
I20251029 08:52:33.363986 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b8f2b34642704cd6bb966b6dcd494d51 (table default.Table_d8be37074ea440ef9fbf1e9d596dbf39 [id=acab1181d22d454abea6817e3481f112]) successfully deleted
I20251029 08:52:33.362700 921 raft_consensus.cc:3060] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:33.364631 2938 tablet_replica.cc:333] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.364820 2938 raft_consensus.cc:2243] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.364955 2938 raft_consensus.cc:2272] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.365445 2938 ts_tablet_manager.cc:1916] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.366139 903 tablet_service.cc:1505] Processing CreateTablet for tablet af576f0e9ccd4c88b8d3c7dd15df1785 (DEFAULT_TABLE table=default.Table_755066834bcd4fc7bf4f0b206ddafc9c [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.366139 1036 tablet_service.cc:1505] Processing CreateTablet for tablet af576f0e9ccd4c88b8d3c7dd15df1785 (DEFAULT_TABLE table=default.Table_755066834bcd4fc7bf4f0b206ddafc9c [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.366556 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af576f0e9ccd4c88b8d3c7dd15df1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.366567 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af576f0e9ccd4c88b8d3c7dd15df1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.363508 2944 tablet.cc:1722] T 23dc7808f8da41e3aa0af5f318cd05c1 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:33.363950 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d26ff78aa5f94ae49a197daf2be00aa5" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.372680 759 tablet_service.cc:1505] Processing CreateTablet for tablet af576f0e9ccd4c88b8d3c7dd15df1785 (DEFAULT_TABLE table=default.Table_755066834bcd4fc7bf4f0b206ddafc9c [id=8ee95d4b3927491c828c62babfa44d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.373277 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af576f0e9ccd4c88b8d3c7dd15df1785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.378489 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 442c37e1ee724b6b96f679f05640baf9 (table default.Table_91a64fd82f164fd8aae36227545d0014 [id=b32652cc709846deb2fc0df05493bafb]). Expected version 1 got 0
I20251029 08:52:33.384709 1177 tablet_bootstrap.cc:654] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.385787 2944 tablet.cc:1722] T 442c37e1ee724b6b96f679f05640baf9 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 08:52:33.386569 2975 mvcc.cc:204] Tried to move back new op lower bound from 7216037689816584192 to 7216037684663521280. Current Snapshot: MvccSnapshot[applied={T|T < 7216037689816584192}]
I20251029 08:52:33.386862 1974 rpcz_store.cc:276] 1029 08:52:26.256830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:26.256881 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:33.330154 (+7073273us) ts_tablet_manager.cc:1151] Deleting tablet c9ed0a270b794e469d2fdbd0ea1d96f8
1029 08:52:33.330164 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.344218 (+ 14054us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.353496 (+ 9278us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.362001 (+ 8505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7073076}
I20251029 08:52:33.388216 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1098) took 6338 ms. Trace:
I20251029 08:52:33.388392 1974 rpcz_store.cc:276] 1029 08:52:27.050087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.050157 (+ 70us) service_pool.cc:225] Handling call
1029 08:52:33.388043 (+6337886us) ts_tablet_manager.cc:1151] Deleting tablet 8c3da4b29e3c4208ae2109789f994c1e
1029 08:52:33.388058 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.388194 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6337691}
I20251029 08:52:33.389292 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1100) took 6332 ms. Trace:
I20251029 08:52:33.392804 1974 rpcz_store.cc:276] 1029 08:52:27.057252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.057302 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:33.389166 (+6331864us) ts_tablet_manager.cc:1151] Deleting tablet d965d5a6b1324b00ad68b28d76796e80
1029 08:52:33.389177 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.389274 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6331675}
I20251029 08:52:33.393347 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1101) took 6334 ms. Trace:
I20251029 08:52:33.393627 2938 ts_tablet_manager.cc:1929] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.396062 1974 rpcz_store.cc:276] 1029 08:52:27.058905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.058954 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:33.393232 (+6334278us) ts_tablet_manager.cc:1151] Deleting tablet af913080bfe44a86a76704ffa32cd9e1
1029 08:52:33.393242 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.393331 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6334120}
I20251029 08:52:33.396565 1974 tablet_replica.cc:333] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.396801 2938 log.cc:1199] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/4e3a2fab467348099a4a1fec74eb5e65
I20251029 08:52:33.396824 1974 raft_consensus.cc:2243] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.397176 1974 raft_consensus.cc:2272] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.397382 2938 ts_tablet_manager.cc:1950] T 4e3a2fab467348099a4a1fec74eb5e65 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.400694 1974 ts_tablet_manager.cc:1916] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.401434 2938 tablet_replica.cc:333] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.401850 2938 raft_consensus.cc:2243] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.402359 2938 pending_rounds.cc:70] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Trying to abort 1 pending ops.
I20251029 08:52:33.402711 2938 pending_rounds.cc:77] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037693851222016 op_type: NO_OP noop_request { }
I20251029 08:52:33.403043 2938 raft_consensus.cc:2889] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:33.403291 2938 raft_consensus.cc:2272] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:33.404434 996 leader_election.cc:343] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.406997 996 leader_election.cc:304] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.407490 2896 raft_consensus.cc:2749] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.410588 2938 ts_tablet_manager.cc:1916] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.411967 882 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50044 (request call id 192) took 1116 ms. Trace:
I20251029 08:52:33.422015 882 rpcz_store.cc:276] 1029 08:52:32.295510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:32.295568 (+ 58us) service_pool.cc:225] Handling call
1029 08:52:33.411934 (+1116366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 08:52:33.423593 1974 ts_tablet_manager.cc:1929] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.427702 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1244) took 2074 ms. Trace:
I20251029 08:52:33.428031 1176 rpcz_store.cc:276] 1029 08:52:31.353560 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.354043 (+ 483us) service_pool.cc:225] Handling call
1029 08:52:33.427568 (+2073525us) ts_tablet_manager.cc:1151] Deleting tablet 8970ca1f227849a1a99ac5d33e450e14
1029 08:52:33.427579 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.427684 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2073394}
I20251029 08:52:33.428355 1974 log.cc:1199] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4aa0ff14e19b4ad7a8b33796481f0f3f
I20251029 08:52:33.428943 1974 ts_tablet_manager.cc:1950] T 4aa0ff14e19b4ad7a8b33796481f0f3f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.429229 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1245) took 2070 ms. Trace:
I20251029 08:52:33.429359 1176 rpcz_store.cc:276] 1029 08:52:31.358961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.359014 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:33.429115 (+2070101us) ts_tablet_manager.cc:1151] Deleting tablet 8a85a1dff3034284a6dcf0e8c3f53297
1029 08:52:33.429126 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.429213 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2069923}
I20251029 08:52:33.430614 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1264) took 1927 ms. Trace:
I20251029 08:52:33.430737 1176 rpcz_store.cc:276] 1029 08:52:31.503156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.508645 (+ 5489us) service_pool.cc:225] Handling call
1029 08:52:33.430514 (+1921869us) ts_tablet_manager.cc:1151] Deleting tablet 39578ce0411646eea7f6d8dc61490e03
1029 08:52:33.430525 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.430599 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921658}
I20251029 08:52:33.430986 1177 tablet_bootstrap.cc:492] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.431162 1177 ts_tablet_manager.cc:1403] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.207s user 0.007s sys 0.003s
I20251029 08:52:33.431268 1176 tablet_replica.cc:333] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.431437 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1105) took 6366 ms. Trace:
I20251029 08:52:33.431706 1176 raft_consensus.cc:2243] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.433966 2938 ts_tablet_manager.cc:1929] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.436318 2938 log.cc:1199] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a9fe3e9a507441b8ba5953c0120289c3
I20251029 08:52:33.437111 1974 rpcz_store.cc:276] 1029 08:52:27.065074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.065140 (+ 66us) service_pool.cc:225] Handling call
1029 08:52:33.396522 (+6331382us) ts_tablet_manager.cc:1151] Deleting tablet 4aa0ff14e19b4ad7a8b33796481f0f3f
1029 08:52:33.396530 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.403917 (+ 7387us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.423464 (+ 19547us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.431420 (+ 7956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6331160}
I20251029 08:52:33.437450 1176 raft_consensus.cc:2272] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.437884 1974 tablet_replica.cc:333] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.438570 1176 ts_tablet_manager.cc:1916] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.431931 1177 raft_consensus.cc:359] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.438661 2938 ts_tablet_manager.cc:1950] T a9fe3e9a507441b8ba5953c0120289c3 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.439013 1177 raft_consensus.cc:385] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.439118 1177 raft_consensus.cc:740] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.438830 1974 raft_consensus.cc:2243] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.439389 1177 consensus_queue.cc:260] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.439975 1177 ts_tablet_manager.cc:1434] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251029 08:52:33.441277 2938 tablet_replica.cc:333] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.441279 1177 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.439570 1974 raft_consensus.cc:2272] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.387663 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c9ed0a270b794e469d2fdbd0ea1d96f8 (table default.Table_b58caa6aed5542948934aeca3e67d32d [id=4614aa88ac7b4d11b1ffe3c19aaee6dc]) successfully deleted
I20251029 08:52:33.398886 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 4e3a2fab467348099a4a1fec74eb5e65 (table default.Table_c68d96e4128e496097a5c5a4740db7ac [id=09dcc91cf1e045169ca0eedfe1e6dad3]) successfully deleted
I20251029 08:52:33.442214 1974 ts_tablet_manager.cc:1916] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.442296 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet e1e2be77182e4259b76d20cae4e9e478 (table default.Table_8b8706e0267f49478503d8a45662ac0b [id=2d957903ec384ca6bba3154fa2438087]) successfully deleted
W20251029 08:52:33.442492 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8970ca1f227849a1a99ac5d33e450e14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8970ca1f227849a1a99ac5d33e450e14 already in progress: creating tablet
W20251029 08:52:33.442638 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 8a85a1dff3034284a6dcf0e8c3f53297 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a85a1dff3034284a6dcf0e8c3f53297 already in progress: creating tablet
W20251029 08:52:33.442763 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 39578ce0411646eea7f6d8dc61490e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39578ce0411646eea7f6d8dc61490e03 already in progress: creating tablet
I20251029 08:52:33.443008 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a9fe3e9a507441b8ba5953c0120289c3 (table default.Table_d74312a6b49e490ab01040a51e1e75e7 [id=d45481f7479f46d3bc3b014c7a437f55]) successfully deleted
W20251029 08:52:33.441932 582 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:42433, user_credentials={real_user=slave}} blocked reactor thread for 54286.2us
W20251029 08:52:33.443543 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8c3da4b29e3c4208ae2109789f994c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3da4b29e3c4208ae2109789f994c1e already in progress: creating tablet
W20251029 08:52:33.443727 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d965d5a6b1324b00ad68b28d76796e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d965d5a6b1324b00ad68b28d76796e80 already in progress: creating tablet
W20251029 08:52:33.443905 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet af913080bfe44a86a76704ffa32cd9e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af913080bfe44a86a76704ffa32cd9e1 already in progress: creating tablet
I20251029 08:52:33.444072 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4aa0ff14e19b4ad7a8b33796481f0f3f (table default.Table_a483eeed2297411daf8f8908b370db56 [id=50ad37b70a054cb1b5fdfb1c940026b9]) successfully deleted
I20251029 08:52:33.446174 1183 tablet_bootstrap.cc:492] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.448623 1183 ts_tablet_manager.cc:1403] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.399s user 0.028s sys 0.033s
I20251029 08:52:33.449486 1183 raft_consensus.cc:359] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.450219 1183 raft_consensus.cc:385] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.450332 1183 raft_consensus.cc:740] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.450598 1183 consensus_queue.cc:260] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.451155 1183 ts_tablet_manager.cc:1434] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:33.452157 1183 tablet_bootstrap.cc:492] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.453974 1177 tablet_bootstrap.cc:654] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.453985 1183 tablet_bootstrap.cc:654] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.459414 2938 raft_consensus.cc:2243] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.460176 2938 raft_consensus.cc:2272] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.461061 2938 ts_tablet_manager.cc:1916] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.463374 1974 ts_tablet_manager.cc:1929] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.463485 1974 log.cc:1199] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/ba201a54466240129d82a096f9316242
I20251029 08:52:33.463840 1974 ts_tablet_manager.cc:1950] T ba201a54466240129d82a096f9316242 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.465868 1176 ts_tablet_manager.cc:1929] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.465901 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 31bed64a27164baa8177b87b54b5947b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e3782c226c14143aec56367d1a04996) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.465981 1176 log.cc:1199] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/9499d0099d1c4edc843fb8d29ca3b1b6
I20251029 08:52:33.466246 1176 ts_tablet_manager.cc:1950] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.468816 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1271) took 1884 ms. Trace:
I20251029 08:52:33.468959 1176 rpcz_store.cc:276] 1029 08:52:31.584635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.584756 (+ 121us) service_pool.cc:225] Handling call
1029 08:52:33.431223 (+1846467us) ts_tablet_manager.cc:1151] Deleting tablet 9499d0099d1c4edc843fb8d29ca3b1b6
1029 08:52:33.431232 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.447680 (+ 16448us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.465767 (+ 18087us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.468796 (+ 3029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846256}
I20251029 08:52:33.469501 1176 tablet_replica.cc:333] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.469760 1176 raft_consensus.cc:2243] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.470080 1176 raft_consensus.cc:2272] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.470284 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 9499d0099d1c4edc843fb8d29ca3b1b6 (table default.Table_c407207334e644189fdbfcae1dc12ba9 [id=e1fb342ea0a541ce9f668a9410db6487]) successfully deleted
I20251029 08:52:33.470750 2920 raft_consensus.cc:493] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.470924 2920 raft_consensus.cc:515] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.471127 1176 ts_tablet_manager.cc:1916] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.471553 2920 leader_election.cc:290] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.472203 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fad7549758649229ad771de926aaf15" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.472791 718 leader_election.cc:343] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.473400 2920 raft_consensus.cc:493] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.473384 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fad7549758649229ad771de926aaf15" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:33.473533 2920 raft_consensus.cc:515] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.474046 2920 leader_election.cc:290] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.474675 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a69beee4032f447eaa694d237cb0e4e3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.475262 720 leader_election.cc:343] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.475365 720 leader_election.cc:304] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.475963 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a69beee4032f447eaa694d237cb0e4e3" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:33.476120 718 leader_election.cc:343] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:33.477918 720 leader_election.cc:343] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.478021 720 leader_election.cc:304] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.478349 2964 raft_consensus.cc:2749] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.478484 2964 raft_consensus.cc:2749] T a69beee4032f447eaa694d237cb0e4e3 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.479238 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1104) took 6414 ms. Trace:
I20251029 08:52:33.480305 1974 rpcz_store.cc:276] 1029 08:52:27.064973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.065207 (+ 234us) service_pool.cc:225] Handling call
1029 08:52:33.437837 (+6372630us) ts_tablet_manager.cc:1151] Deleting tablet ba201a54466240129d82a096f9316242
1029 08:52:33.437847 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.450632 (+ 12785us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.463276 (+ 12644us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.479219 (+ 15943us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14288,"tablet-delete.queue_time_us":6372464}
I20251029 08:52:33.481163 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1103) took 6416 ms. Trace:
I20251029 08:52:33.481253 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet ba201a54466240129d82a096f9316242 (table default.Table_9c7a9b9d1c70476ebfe5556126bb0707 [id=fef9c26e57d3498a84bd2a056aec2911]) successfully deleted
I20251029 08:52:33.481802 1974 rpcz_store.cc:276] 1029 08:52:27.064857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.065320 (+ 463us) service_pool.cc:225] Handling call
1029 08:52:33.481052 (+6415732us) ts_tablet_manager.cc:1151] Deleting tablet 58932f38c0d64aec9a02238eaea56a25
1029 08:52:33.481061 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.481149 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6415340}
W20251029 08:52:33.482609 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 58932f38c0d64aec9a02238eaea56a25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58932f38c0d64aec9a02238eaea56a25 already in progress: creating tablet
I20251029 08:52:33.482561 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1113) took 6297 ms. Trace:
I20251029 08:52:33.483398 1974 rpcz_store.cc:276] 1029 08:52:27.184756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.184853 (+ 97us) service_pool.cc:225] Handling call
1029 08:52:33.482463 (+6297610us) ts_tablet_manager.cc:1151] Deleting tablet 2fad7549758649229ad771de926aaf15
1029 08:52:33.482472 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.482547 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6296849}
I20251029 08:52:33.484073 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1111) took 6330 ms. Trace:
W20251029 08:52:33.484194 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 2fad7549758649229ad771de926aaf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fad7549758649229ad771de926aaf15 already in progress: creating tablet
I20251029 08:52:33.486249 2938 ts_tablet_manager.cc:1929] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.486800 1974 rpcz_store.cc:276] 1029 08:52:27.153367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.239727 (+ 86360us) service_pool.cc:225] Handling call
1029 08:52:33.483967 (+6244240us) ts_tablet_manager.cc:1151] Deleting tablet bd357a723a1d4bc1aa4b480c8a29ef03
1029 08:52:33.483975 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.484060 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6244035}
I20251029 08:52:33.488972 1176 ts_tablet_manager.cc:1929] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:33.492638 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bd357a723a1d4bc1aa4b480c8a29ef03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd357a723a1d4bc1aa4b480c8a29ef03 already in progress: creating tablet
I20251029 08:52:33.493544 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1117) took 6217 ms. Trace:
I20251029 08:52:33.493631 1974 rpcz_store.cc:276] 1029 08:52:27.275872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.275926 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:33.492965 (+6217039us) ts_tablet_manager.cc:1151] Deleting tablet cea6d85081f246788ba29bb4a9c9a4e8
1029 08:52:33.492974 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.493071 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6216833}
I20251029 08:52:33.493747 1176 log.cc:1199] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/42de32d87ebb451090e0310e36e64277
W20251029 08:52:33.493995 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet cea6d85081f246788ba29bb4a9c9a4e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea6d85081f246788ba29bb4a9c9a4e8 already in progress: creating tablet
I20251029 08:52:33.494402 1176 ts_tablet_manager.cc:1950] T 42de32d87ebb451090e0310e36e64277 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.494415 595 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_a5ebbce3f0ef404699dae9a0c1c670c3 [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 2 got 0
I20251029 08:52:33.494860 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1116) took 6219 ms. Trace:
I20251029 08:52:33.494936 1974 rpcz_store.cc:276] 1029 08:52:27.275608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.276210 (+ 602us) service_pool.cc:225] Handling call
1029 08:52:33.494762 (+6218552us) ts_tablet_manager.cc:1151] Deleting tablet f744556dccf348fd8b92071f224730aa
1029 08:52:33.494771 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.494847 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6218315}
W20251029 08:52:33.495256 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f744556dccf348fd8b92071f224730aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f744556dccf348fd8b92071f224730aa already in progress: creating tablet
I20251029 08:52:33.495607 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1118) took 6217 ms. Trace:
I20251029 08:52:33.495604 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1270) took 1963 ms. Trace:
I20251029 08:52:33.495680 1974 rpcz_store.cc:276] 1029 08:52:27.277785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.277848 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:33.495521 (+6217673us) ts_tablet_manager.cc:1151] Deleting tablet 38032950744b4426927157fad0962dfb
1029 08:52:33.495529 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.495594 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6217517}
I20251029 08:52:33.495694 1176 rpcz_store.cc:276] 1029 08:52:31.531888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.611995 (+ 80107us) service_pool.cc:225] Handling call
1029 08:52:33.469454 (+1857459us) ts_tablet_manager.cc:1151] Deleting tablet 42de32d87ebb451090e0310e36e64277
1029 08:52:33.469464 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.474231 (+ 4767us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.488884 (+ 14653us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.495579 (+ 6695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1857240}
W20251029 08:52:33.496001 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 38032950744b4426927157fad0962dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38032950744b4426927157fad0962dfb already in progress: creating tablet
I20251029 08:52:33.496367 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1119) took 6203 ms. Trace:
I20251029 08:52:33.496479 1974 rpcz_store.cc:276] 1029 08:52:27.292936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.295176 (+ 2240us) service_pool.cc:225] Handling call
1029 08:52:33.496276 (+6201100us) ts_tablet_manager.cc:1151] Deleting tablet 46bcc468ce964610a857bc2b11b5b3e6
1029 08:52:33.496285 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.496353 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6200855}
I20251029 08:52:33.496750 2938 log.cc:1199] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/85df43bf781c421688f163331de0eb2b
W20251029 08:52:33.497068 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 46bcc468ce964610a857bc2b11b5b3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bcc468ce964610a857bc2b11b5b3e6 already in progress: creating tablet
I20251029 08:52:33.497393 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1121) took 6171 ms. Trace:
I20251029 08:52:33.497792 2938 ts_tablet_manager.cc:1950] T 85df43bf781c421688f163331de0eb2b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.497901 1974 rpcz_store.cc:276] 1029 08:52:27.326270 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.326334 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:33.497319 (+6170985us) ts_tablet_manager.cc:1151] Deleting tablet e687f593eead488db4bb445a63beeec7
1029 08:52:33.497327 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.497378 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6170780}
W20251029 08:52:33.498317 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e687f593eead488db4bb445a63beeec7 because the tablet was not found. No further retry: Not found: Tablet not found: e687f593eead488db4bb445a63beeec7
I20251029 08:52:33.498611 1974 tablet_replica.cc:333] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.499630 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 85df43bf781c421688f163331de0eb2b (table default.Table_7501d57fa16e41269af41cad333961d2 [id=2e09778b9b214426bdcd4070c9f0abca]) successfully deleted
I20251029 08:52:33.499986 1974 raft_consensus.cc:2243] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.500629 1974 raft_consensus.cc:2272] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.500736 1177 tablet_bootstrap.cc:492] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.500921 1177 ts_tablet_manager.cc:1403] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.005s
I20251029 08:52:33.501176 1974 ts_tablet_manager.cc:1916] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.496672 1178 tablet_bootstrap.cc:492] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.501329 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 42de32d87ebb451090e0310e36e64277 (table default.Table_007a17f442c74a71a6bb3775642f35fd [id=3ca17547257d4c358a173ebef3ef4a43]) successfully deleted
I20251029 08:52:33.501371 1178 ts_tablet_manager.cc:1403] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.306s user 0.043s sys 0.046s
W20251029 08:52:33.501910 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 9f8348b6234e4dbe8bdb58edad34ea7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f8348b6234e4dbe8bdb58edad34ea7b already in progress: creating tablet
I20251029 08:52:33.502188 2938 tablet_replica.cc:333] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.502377 2938 raft_consensus.cc:2243] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.502528 1178 raft_consensus.cc:359] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.502657 1178 raft_consensus.cc:385] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.502714 1178 raft_consensus.cc:740] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.502753 2938 raft_consensus.cc:2272] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.502914 1178 consensus_queue.cc:260] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.503394 1178 ts_tablet_manager.cc:1434] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:33.503775 2938 ts_tablet_manager.cc:1916] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.504206 1177 raft_consensus.cc:359] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.504431 1177 raft_consensus.cc:385] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.504631 1177 raft_consensus.cc:740] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.504879 1177 consensus_queue.cc:260] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.507838 1177 ts_tablet_manager.cc:1434] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251029 08:52:33.508631 1183 tablet_bootstrap.cc:492] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.509573 1183 ts_tablet_manager.cc:1403] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.010s
I20251029 08:52:33.512064 588 master_service.cc:954] ReplaceTablet: received request to replace tablet d3955ff79faf42bb82f764aae79b85d9 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:33.512147 1178 tablet_bootstrap.cc:492] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.512603 1183 raft_consensus.cc:359] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.512874 1183 raft_consensus.cc:385] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.512998 1183 raft_consensus.cc:740] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.513275 1183 consensus_queue.cc:260] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.513537 1178 tablet_bootstrap.cc:654] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.517179 1183 ts_tablet_manager.cc:1434] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251029 08:52:33.517526 1183 tablet_bootstrap.cc:492] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.519182 1183 tablet_bootstrap.cc:654] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.522447 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1273) took 1937 ms. Trace:
I20251029 08:52:33.522603 1176 rpcz_store.cc:276] 1029 08:52:31.584828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.704421 (+119593us) service_pool.cc:225] Handling call
1029 08:52:33.503553 (+1799132us) ts_tablet_manager.cc:1151] Deleting tablet adb59d3ee338465a942e6b4423f0e071
1029 08:52:33.522322 (+ 18769us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.522428 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798909}
W20251029 08:52:33.523100 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet adb59d3ee338465a942e6b4423f0e071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adb59d3ee338465a942e6b4423f0e071 already in progress: creating tablet
I20251029 08:52:33.523459 1176 tablet_replica.cc:333] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.523697 1176 raft_consensus.cc:2243] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.523883 1176 raft_consensus.cc:2272] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.524487 1176 ts_tablet_manager.cc:1916] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.532869 2920 raft_consensus.cc:493] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.533010 2920 raft_consensus.cc:515] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.533567 2920 leader_election.cc:290] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.534261 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65eaf12e45c54bd1b0aedc647dc90436" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:33.534641 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65eaf12e45c54bd1b0aedc647dc90436" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.535274 1176 ts_tablet_manager.cc:1929] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.535363 1176 log.cc:1199] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/485c9946f3f14a2aab11f4cdaa70a5b6
W20251029 08:52:33.535395 718 leader_election.cc:343] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.535642 1176 ts_tablet_manager.cc:1950] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.536276 1183 tablet_bootstrap.cc:492] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.536360 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39982 (request call id 1246) took 2093 ms. Trace:
I20251029 08:52:33.536491 1183 ts_tablet_manager.cc:1403] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
W20251029 08:52:33.536595 720 leader_election.cc:343] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.536499 1176 rpcz_store.cc:276] 1029 08:52:31.442862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.758487 (+315625us) service_pool.cc:225] Handling call
1029 08:52:33.523417 (+1764930us) ts_tablet_manager.cc:1151] Deleting tablet 485c9946f3f14a2aab11f4cdaa70a5b6
1029 08:52:33.523426 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.532221 (+ 8795us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.535196 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.536342 (+ 1146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764704}
I20251029 08:52:33.536706 720 leader_election.cc:304] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.536911 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 485c9946f3f14a2aab11f4cdaa70a5b6 (table default.Table_5df6707d7a754de3b61f32cf68786b2e [id=2f663ee400904f1d9e21d15b83c58e24]) successfully deleted
I20251029 08:52:33.537209 2920 raft_consensus.cc:2749] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.537315 1183 raft_consensus.cc:359] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.537544 1183 raft_consensus.cc:385] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.537659 1183 raft_consensus.cc:740] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:33.537750 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 82589b43556c46559f0de4af7e8d9db0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82589b43556c46559f0de4af7e8d9db0 already in progress: creating tablet
I20251029 08:52:33.537943 1183 consensus_queue.cc:260] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
W20251029 08:52:33.538358 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet ac09bb4502e440368633bbc90e9ad2cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac09bb4502e440368633bbc90e9ad2cf already in progress: creating tablet
I20251029 08:52:33.538503 1183 ts_tablet_manager.cc:1434] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:33.538753 1183 tablet_bootstrap.cc:492] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
W20251029 08:52:33.538970 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet a17142ffce3e4c359e715ef63747c40d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17142ffce3e4c359e715ef63747c40d already in progress: creating tablet
W20251029 08:52:33.539522 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet d010353b251140e99ce2df996f965991 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d010353b251140e99ce2df996f965991 already in progress: creating tablet
I20251029 08:52:33.539813 1176 tablet_replica.cc:333] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.540427 1176 raft_consensus.cc:2243] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.540792 1176 raft_consensus.cc:2272] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.541517 1176 ts_tablet_manager.cc:1916] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.548331 1178 tablet_bootstrap.cc:492] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.548455 1178 ts_tablet_manager.cc:1403] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251029 08:52:33.549167 1178 raft_consensus.cc:359] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.549296 1183 tablet_bootstrap.cc:654] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.549312 1178 raft_consensus.cc:385] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.549363 1178 raft_consensus.cc:740] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.549427 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a506e35afe6a4c87901f27c434f3b348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:13 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.549604 1178 consensus_queue.cc:260] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.550078 1178 ts_tablet_manager.cc:1434] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 08:52:33.550767 1178 tablet_bootstrap.cc:492] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.551136 1974 ts_tablet_manager.cc:1929] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.551256 1974 log.cc:1199] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aa4f97075b3941aaa7cda3f9413e867f
I20251029 08:52:33.551638 1974 ts_tablet_manager.cc:1950] T aa4f97075b3941aaa7cda3f9413e867f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.552433 1178 tablet_bootstrap.cc:654] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.552495 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1122) took 6225 ms. Trace:
I20251029 08:52:33.552589 1974 rpcz_store.cc:276] 1029 08:52:27.326826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.326878 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:33.498570 (+6171692us) ts_tablet_manager.cc:1151] Deleting tablet aa4f97075b3941aaa7cda3f9413e867f
1029 08:52:33.498580 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.507253 (+ 8673us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.531941 (+ 24688us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.552472 (+ 20531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6171518}
I20251029 08:52:33.553023 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet aa4f97075b3941aaa7cda3f9413e867f (table default.Table_4c3334f5e9ce4296866318c8b7fbe1eb [id=1b80fa2360644c0fb6f69660b381e005]) successfully deleted
I20251029 08:52:33.554116 2938 ts_tablet_manager.cc:1929] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.554236 2938 log.cc:1199] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d26ff78aa5f94ae49a197daf2be00aa5
I20251029 08:52:33.554533 2938 ts_tablet_manager.cc:1950] T d26ff78aa5f94ae49a197daf2be00aa5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.555420 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d26ff78aa5f94ae49a197daf2be00aa5 (table default.Table_5018010ef9af4f16b6388715c7f9fd03 [id=edda8e15b59b472886bb1eba8fec8739]) successfully deleted
I20251029 08:52:33.555704 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 with delete_type TABLET_DATA_DELETED (replaced by tablet 295ab2a10a5648abb59bc045c7dbe165) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.555744 1974 tablet_replica.cc:333] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.555948 1974 raft_consensus.cc:2243] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.556134 1974 raft_consensus.cc:2272] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:33.557171 584 catalog_manager.cc:4952] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 2fafe9a0731b48b3a60032d94623681d because the tablet was not found. No further retry: Not found: Tablet not found: 2fafe9a0731b48b3a60032d94623681d
I20251029 08:52:33.557448 2938 tablet_replica.cc:333] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.557641 2938 raft_consensus.cc:2243] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.557793 2938 raft_consensus.cc:2272] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.558297 2938 ts_tablet_manager.cc:1916] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.560679 1974 ts_tablet_manager.cc:1916] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.561061 2920 raft_consensus.cc:493] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.561187 2920 raft_consensus.cc:515] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.561764 2920 leader_election.cc:290] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:33.562371 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85a806cc04e641d9a1c4af97a24b13de" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:33.562455 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85a806cc04e641d9a1c4af97a24b13de" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.562790 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 59c90087c9e94665a9b3fac798304af7 (table default.Table_645b4972920f4607880280733ebe04a6 [id=5b45ec162f5e43d4a7976bb7f6426dbd]). Expected version 1 got 0
W20251029 08:52:33.562804 720 leader_election.cc:343] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:33.563493 718 leader_election.cc:343] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.563598 718 leader_election.cc:304] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.563978 2920 raft_consensus.cc:2749] T 85a806cc04e641d9a1c4af97a24b13de P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.566265 759 tablet_service.cc:1552] Processing DeleteTablet for tablet d0d00a6b772d4034ac7fa764884c7081 with delete_type TABLET_DATA_DELETED (replaced by tablet e57e4313adce4e35843ccd9ee137e8f7) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.571460 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a355c68ecfb45ff8c396873d91268a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.572834 1176 ts_tablet_manager.cc:1929] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.572939 1176 log.cc:1199] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/3cac145320cf48b4aaf4adb0b686ba8c
I20251029 08:52:33.573211 1176 ts_tablet_manager.cc:1950] T 3cac145320cf48b4aaf4adb0b686ba8c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.578984 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 3cac145320cf48b4aaf4adb0b686ba8c (table default.Table_43e66e8dbdf240a1b5eb4233ece857c7 [id=26886277edf44c6cbffbd904706dc430]) successfully deleted
I20251029 08:52:33.579622 1176 tablet_replica.cc:333] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.582635 2938 ts_tablet_manager.cc:1929] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.583115 1974 ts_tablet_manager.cc:1929] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.583472 1974 log.cc:1199] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/aa0ef904c2fb4dfeadbf56ae659c4e20
I20251029 08:52:33.583925 1974 ts_tablet_manager.cc:1950] T aa0ef904c2fb4dfeadbf56ae659c4e20 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.583995 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c5525b69c8a4450b6d6571d39abb079 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:33.584362 2938 log.cc:1199] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/62597189ecb047788b83788b514e9099
I20251029 08:52:33.585165 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1123) took 6257 ms. Trace:
I20251029 08:52:33.585388 1176 raft_consensus.cc:2243] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.585578 1176 raft_consensus.cc:2272] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.585857 2938 ts_tablet_manager.cc:1950] T 62597189ecb047788b83788b514e9099 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.586086 1176 ts_tablet_manager.cc:1916] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.588193 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 62597189ecb047788b83788b514e9099 (table default.Table_6056be95de0a4b34ab9dcf62b2968444 [id=c14f386ee208443db44d014c6b613070]) successfully deleted
I20251029 08:52:33.588596 1974 rpcz_store.cc:276] 1029 08:52:27.327570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.327616 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:33.555689 (+6228073us) ts_tablet_manager.cc:1151] Deleting tablet aa0ef904c2fb4dfeadbf56ae659c4e20
1029 08:52:33.555701 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.565554 (+ 9853us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.583029 (+ 17475us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.585148 (+ 2119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6227906}
I20251029 08:52:33.589187 2938 tablet_replica.cc:333] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.589435 2938 raft_consensus.cc:2243] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.589764 2938 raft_consensus.cc:2272] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.590516 2938 ts_tablet_manager.cc:1916] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.591367 1183 tablet_bootstrap.cc:492] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.591531 1183 ts_tablet_manager.cc:1403] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.004s
I20251029 08:52:33.592366 1183 raft_consensus.cc:359] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.592592 1183 raft_consensus.cc:385] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.592695 1183 raft_consensus.cc:740] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.592989 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet aa0ef904c2fb4dfeadbf56ae659c4e20 (table default.Table_57fcb0952cf24438aadd4476f7f120a7 [id=377cfe2bd14d4095935f5de04de5db0d]) successfully deleted
I20251029 08:52:33.592957 1183 consensus_queue.cc:260] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.593169 1178 tablet_bootstrap.cc:492] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.593304 1178 ts_tablet_manager.cc:1403] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20251029 08:52:33.593958 1178 raft_consensus.cc:359] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.594094 1178 raft_consensus.cc:385] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.594136 1178 raft_consensus.cc:740] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.594300 1178 consensus_queue.cc:260] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.594871 1178 ts_tablet_manager.cc:1434] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 08:52:33.595124 1178 tablet_bootstrap.cc:492] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.596438 1178 tablet_bootstrap.cc:654] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.597144 1183 ts_tablet_manager.cc:1434] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 08:52:33.597580 1183 tablet_bootstrap.cc:492] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.601400 1176 ts_tablet_manager.cc:1929] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:33.601500 1176 log.cc:1199] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2fafe9a0731b48b3a60032d94623681d
I20251029 08:52:33.602811 1176 ts_tablet_manager.cc:1950] T 2fafe9a0731b48b3a60032d94623681d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.611115 1974 tablet_replica.cc:333] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.611398 1974 raft_consensus.cc:2243] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.611828 1974 raft_consensus.cc:2272] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.612478 1974 ts_tablet_manager.cc:1916] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.616214 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 34729c3f97a44713a352bde297a80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.617609 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2fafe9a0731b48b3a60032d94623681d (table default.Table_5f0752745f5b45c7a3e2be18e3fac5a9 [id=683a0914751f4a4ca50cf1d4588ca225]) successfully deleted
I20251029 08:52:33.617897 1176 tablet_replica.cc:333] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.618068 1176 raft_consensus.cc:2243] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.618193 1176 raft_consensus.cc:2272] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.618700 1176 ts_tablet_manager.cc:1916] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.622625 759 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.624728 2920 raft_consensus.cc:493] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.624861 2920 raft_consensus.cc:515] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.625463 2920 leader_election.cc:290] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.625801 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00829c7fa0fe4e379a1a3044550e5941" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:33.626200 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00829c7fa0fe4e379a1a3044550e5941" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.626672 718 leader_election.cc:343] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.626760 1176 ts_tablet_manager.cc:1929] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:33.626796 720 leader_election.cc:343] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.626863 1176 log.cc:1199] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/308da528313044929b66040d6659f774
I20251029 08:52:33.626914 720 leader_election.cc:304] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.627139 1176 ts_tablet_manager.cc:1950] T 308da528313044929b66040d6659f774 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.627362 2920 raft_consensus.cc:2749] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.628087 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 308da528313044929b66040d6659f774 (table default.Table_9f4e595ea8564e9d902458aff13d5c78 [id=205b9176bb44494b9b2f384073458db8]) successfully deleted
W20251029 08:52:33.629016 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 1b3ab5946c574171b66f206c85720f79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b3ab5946c574171b66f206c85720f79 already in progress: creating tablet
I20251029 08:52:33.629123 1176 tablet_replica.cc:333] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.629293 1176 raft_consensus.cc:2243] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.629417 1176 raft_consensus.cc:2272] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:33.629653 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet cb1e2c24e30048abb3ae9fb92171e049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb1e2c24e30048abb3ae9fb92171e049 already in progress: creating tablet
W20251029 08:52:33.629843 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet f8a896408fd142b9b41d606738234ba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a896408fd142b9b41d606738234ba4 already in progress: creating tablet
I20251029 08:52:33.629910 1176 ts_tablet_manager.cc:1916] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.631608 1183 tablet_bootstrap.cc:654] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.635231 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 7c5525b69c8a4450b6d6571d39abb079
I20251029 08:52:33.635836 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 with delete_type TABLET_DATA_DELETED (replaced by tablet 02cf976c0f164a708ca80ed653319853) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.636324 588 catalog_manager.cc:6719] ReplaceTablet: tablet 7c5525b69c8a4450b6d6571d39abb079 deleted and replaced by tablet 02cf976c0f164a708ca80ed653319853
I20251029 08:52:33.640537 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 with delete_type TABLET_DATA_DELETED (replaced by tablet 02cf976c0f164a708ca80ed653319853) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.641072 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c5525b69c8a4450b6d6571d39abb079 with delete_type TABLET_DATA_DELETED (replaced by tablet 02cf976c0f164a708ca80ed653319853) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.646773 1176 ts_tablet_manager.cc:1929] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.646896 1176 log.cc:1199] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/5f67e7b28b4240618e1817bb47d5633d
I20251029 08:52:33.647187 1176 ts_tablet_manager.cc:1950] T 5f67e7b28b4240618e1817bb47d5633d P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.648093 1176 tablet_replica.cc:333] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.648231 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 5f67e7b28b4240618e1817bb47d5633d (table default.Table_f38c5c596ef148eb8ac7ea611a4e4b3d [id=66eee56ac86141938732092093b9e0b2]) successfully deleted
I20251029 08:52:33.648277 1176 raft_consensus.cc:2243] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.648413 1176 raft_consensus.cc:2272] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.648906 1176 ts_tablet_manager.cc:1916] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.650794 1974 ts_tablet_manager.cc:1929] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.652516 2896 raft_consensus.cc:493] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.652693 2896 raft_consensus.cc:515] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.653504 2896 leader_election.cc:290] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:33.653956 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34729c3f97a44713a352bde297a80027" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.654536 998 leader_election.cc:343] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 34729c3f97a44713a352bde297a80027
W20251029 08:52:33.654655 996 leader_election.cc:343] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.654772 996 leader_election.cc:304] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.655225 2896 raft_consensus.cc:2749] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.654639 1974 log.cc:1199] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/3c1a8a1e4cc14f1aa6e4f91174f45127
I20251029 08:52:33.655870 1974 ts_tablet_manager.cc:1950] T 3c1a8a1e4cc14f1aa6e4f91174f45127 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.656492 1178 tablet_bootstrap.cc:492] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.656611 1178 ts_tablet_manager.cc:1403] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.011s
I20251029 08:52:33.657251 1178 raft_consensus.cc:359] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.657393 1178 raft_consensus.cc:385] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.657437 1178 raft_consensus.cc:740] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.657632 1178 consensus_queue.cc:260] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.658090 1178 ts_tablet_manager.cc:1434] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:33.658358 1178 tablet_bootstrap.cc:492] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.659709 1178 tablet_bootstrap.cc:654] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.664783 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1120) took 6339 ms. Trace:
I20251029 08:52:33.665151 1974 rpcz_store.cc:276] 1029 08:52:27.325690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.328068 (+ 2378us) service_pool.cc:225] Handling call
1029 08:52:33.589032 (+6260964us) ts_tablet_manager.cc:1151] Deleting tablet 3c1a8a1e4cc14f1aa6e4f91174f45127
1029 08:52:33.611052 (+ 22020us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.616108 (+ 5056us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.650691 (+ 34583us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.664760 (+ 14069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6260770}
I20251029 08:52:33.666855 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 38032950744b4426927157fad0962dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:27 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.668936 1176 ts_tablet_manager.cc:1929] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.668972 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 3c1a8a1e4cc14f1aa6e4f91174f45127 (table default.Table_3a2964132be042848bb5e8b54e2274d1 [id=249248f9ea8e4f23bac7bda9e1f53f47]) successfully deleted
I20251029 08:52:33.669044 1176 log.cc:1199] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2f36641b7086430a89380056e04b8aaf
I20251029 08:52:33.669334 1176 ts_tablet_manager.cc:1950] T 2f36641b7086430a89380056e04b8aaf P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.669529 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1127) took 6232 ms. Trace:
I20251029 08:52:33.672751 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2f36641b7086430a89380056e04b8aaf (table default.Table_4cee30efd10f47c88ab3c9c5a2df5348 [id=bb572b68215b450c9417191114794b24]) successfully deleted
I20251029 08:52:33.672947 1974 rpcz_store.cc:276] 1029 08:52:27.436945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.436999 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:33.669348 (+6232349us) ts_tablet_manager.cc:1151] Deleting tablet 960f17ab6bed4ef5ad382421d84bf8d7
1029 08:52:33.669358 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.669509 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6232156}
W20251029 08:52:33.673959 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet c7818ad8e5ec4fca8e3b25019f545dd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7818ad8e5ec4fca8e3b25019f545dd8 already in progress: creating tablet
W20251029 08:52:33.674609 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 960f17ab6bed4ef5ad382421d84bf8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960f17ab6bed4ef5ad382421d84bf8d7 already in progress: creating tablet
I20251029 08:52:33.674875 1178 tablet_bootstrap.cc:492] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.674998 1178 ts_tablet_manager.cc:1403] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20251029 08:52:33.675129 1974 tablet_replica.cc:333] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.675609 1178 raft_consensus.cc:359] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.675797 1178 raft_consensus.cc:385] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.675854 1178 raft_consensus.cc:740] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.672633 1177 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.676091 1178 consensus_queue.cc:260] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.676162 1974 raft_consensus.cc:2243] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.676533 1178 ts_tablet_manager.cc:1434] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251029 08:52:33.676797 1974 pending_rounds.cc:70] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:33.677037 1178 tablet_bootstrap.cc:492] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.678717 1176 tablet_replica.cc:333] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.678910 1176 raft_consensus.cc:2243] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.679049 1176 raft_consensus.cc:2272] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.679534 1176 ts_tablet_manager.cc:1916] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.679031 1974 pending_rounds.cc:77] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037675268792320 op_type: NO_OP noop_request { }
I20251029 08:52:33.679749 1974 raft_consensus.cc:2889] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:33.679903 1974 raft_consensus.cc:2272] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.680511 1974 ts_tablet_manager.cc:1916] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.685485 1177 tablet_bootstrap.cc:654] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.685632 1176 ts_tablet_manager.cc:1929] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.685737 1176 log.cc:1199] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d444a5d489434696ae178c6d650bf8d5
I20251029 08:52:33.686038 1176 ts_tablet_manager.cc:1950] T d444a5d489434696ae178c6d650bf8d5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.686908 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d444a5d489434696ae178c6d650bf8d5 (table default.Table_55491b160d0e4e6185be2e0fee2e67a5 [id=8431eac1a49a41c080964575e16345d7]) successfully deleted
I20251029 08:52:33.687090 1178 tablet_bootstrap.cc:654] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.688241 2938 ts_tablet_manager.cc:1929] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.688366 2938 log.cc:1199] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/31bed64a27164baa8177b87b54b5947b
I20251029 08:52:33.688663 2938 ts_tablet_manager.cc:1950] T 31bed64a27164baa8177b87b54b5947b P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:33.689543 2938 tablet_replica.cc:333] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:33.689725 2938 raft_consensus.cc:2243] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.689881 2938 raft_consensus.cc:2272] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.690716 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 31bed64a27164baa8177b87b54b5947b (table default.Table_2134798e3d764329b406d0ea336c2cab [id=837bd17fc74b4c2ab8b82942e15f8be4]) successfully deleted
I20251029 08:52:33.691565 1178 tablet_bootstrap.cc:492] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.691687 1178 ts_tablet_manager.cc:1403] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251029 08:52:33.692785 1183 tablet_bootstrap.cc:492] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.692925 1183 ts_tablet_manager.cc:1403] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.002s
I20251029 08:52:33.693626 1183 raft_consensus.cc:359] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.693810 1183 raft_consensus.cc:385] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.693883 1183 raft_consensus.cc:740] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.694367 1183 consensus_queue.cc:260] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.694839 1178 raft_consensus.cc:359] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.694983 1178 raft_consensus.cc:385] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.695026 1178 raft_consensus.cc:740] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.698498 1183 ts_tablet_manager.cc:1434] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 08:52:33.698742 1178 consensus_queue.cc:260] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.699402 1178 ts_tablet_manager.cc:1434] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 08:52:33.699947 1178 tablet_bootstrap.cc:492] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.700641 1177 tablet_bootstrap.cc:492] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.700778 1177 ts_tablet_manager.cc:1403] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20251029 08:52:33.701793 1178 tablet_bootstrap.cc:654] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:33.702942 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 9f8348b6234e4dbe8bdb58edad34ea7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f8348b6234e4dbe8bdb58edad34ea7b already in progress: creating tablet
I20251029 08:52:33.703249 1176 tablet_replica.cc:333] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.703662 1177 raft_consensus.cc:359] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.703835 1177 raft_consensus.cc:385] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.703903 1177 raft_consensus.cc:740] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.704166 1177 consensus_queue.cc:260] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.703423 1176 raft_consensus.cc:2243] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.705471 1176 raft_consensus.cc:2272] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.705958 1176 ts_tablet_manager.cc:1916] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.710935 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d6d720fb6b54b1e8bb7b3956c48b3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.711174 1183 tablet_bootstrap.cc:492] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.712929 1177 ts_tablet_manager.cc:1434] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251029 08:52:33.713042 1183 tablet_bootstrap.cc:654] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.715728 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet d5bb0fd4c62d47a2b10719ebd55851c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.725368 1177 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.726317 1176 ts_tablet_manager.cc:1929] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.726442 1176 log.cc:1199] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/2fad7549758649229ad771de926aaf15
I20251029 08:52:33.726748 1176 ts_tablet_manager.cc:1950] T 2fad7549758649229ad771de926aaf15 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.727224 1177 tablet_bootstrap.cc:654] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.727803 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 2fad7549758649229ad771de926aaf15 (table default.Table_4a8284ff6d314bbc976db77d1cb42d29 [id=5a81245c63184c9b9f5246f17d6e3a4e]) successfully deleted
I20251029 08:52:33.730063 1974 ts_tablet_manager.cc:1929] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.730247 1974 log.cc:1199] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/24834985a71349b9865840e9b5537d72
I20251029 08:52:33.730660 1974 ts_tablet_manager.cc:1950] T 24834985a71349b9865840e9b5537d72 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.731221 2938 ts_tablet_manager.cc:1916] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.731662 1176 tablet_replica.cc:333] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.731592 2961 raft_consensus.cc:493] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.731860 1176 raft_consensus.cc:2243] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.731827 2961 raft_consensus.cc:515] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.731997 1176 raft_consensus.cc:2272] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.732435 2961 leader_election.cc:290] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:33.732481 1176 ts_tablet_manager.cc:1916] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:33.734081 861 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 254e62d88c3242a0a3ac7470a8005d74
W20251029 08:52:33.734274 861 leader_election.cc:343] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:33.734345 861 leader_election.cc:304] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:33.734808 2961 raft_consensus.cc:2749] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.735774 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1128) took 6279 ms. Trace:
I20251029 08:52:33.731519 1183 tablet_bootstrap.cc:492] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.736776 1183 ts_tablet_manager.cc:1403] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
I20251029 08:52:33.737120 1178 tablet_bootstrap.cc:492] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.737244 1178 ts_tablet_manager.cc:1403] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.001s
I20251029 08:52:33.737790 1183 raft_consensus.cc:359] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.737849 1178 raft_consensus.cc:359] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.738000 1178 raft_consensus.cc:385] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.738056 1178 raft_consensus.cc:740] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.738160 1183 raft_consensus.cc:385] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.738250 1178 consensus_queue.cc:260] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.738715 1178 ts_tablet_manager.cc:1434] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:33.739538 1178 tablet_bootstrap.cc:492] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.738353 1183 raft_consensus.cc:740] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.741227 1183 consensus_queue.cc:260] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.741818 1183 ts_tablet_manager.cc:1434] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 08:52:33.743706 1178 tablet_bootstrap.cc:654] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.745469 1183 tablet_bootstrap.cc:492] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.747442 1183 tablet_bootstrap.cc:654] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.736366 1177 tablet_bootstrap.cc:492] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.746158 2961 raft_consensus.cc:493] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.751050 1177 ts_tablet_manager.cc:1403] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20251029 08:52:33.751073 2961 raft_consensus.cc:515] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.751832 2961 leader_election.cc:290] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
I20251029 08:52:33.751885 1177 raft_consensus.cc:359] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.752050 1177 raft_consensus.cc:385] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.752105 1177 raft_consensus.cc:740] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.752305 1177 consensus_queue.cc:260] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.752722 1177 ts_tablet_manager.cc:1434] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251029 08:52:33.752997 1177 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:33.753592 861 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 6d6d720fb6b54b1e8bb7b3956c48b3f9
W20251029 08:52:33.753800 861 leader_election.cc:343] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 6d6d720fb6b54b1e8bb7b3956c48b3f9
I20251029 08:52:33.753880 861 leader_election.cc:304] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:33.754261 1177 tablet_bootstrap.cc:654] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.754279 2972 raft_consensus.cc:2749] T 6d6d720fb6b54b1e8bb7b3956c48b3f9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.750931 1974 rpcz_store.cc:276] 1029 08:52:27.456655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.456719 (+ 64us) service_pool.cc:225] Handling call
1029 08:52:33.675019 (+6218300us) ts_tablet_manager.cc:1151] Deleting tablet 24834985a71349b9865840e9b5537d72
1029 08:52:33.675033 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.720126 (+ 45093us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.729965 (+ 9839us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.735756 (+ 5791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6218126}
I20251029 08:52:33.755029 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 24834985a71349b9865840e9b5537d72 (table default.Table_0b00cd7c457d4204965de864c88be5e6 [id=0158571f3f9246aeafbfd8b8b1548482]) successfully deleted
I20251029 08:52:33.758718 903 tablet_service.cc:1552] Processing DeleteTablet for tablet e2b87d133f8a4bd1877e07a1a8758147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.759110 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1138) took 6132 ms. Trace:
I20251029 08:52:33.764331 1974 rpcz_store.cc:276] 1029 08:52:27.626391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.630771 (+ 4380us) service_pool.cc:225] Handling call
1029 08:52:33.755355 (+6124584us) ts_tablet_manager.cc:1151] Deleting tablet 21d409c2e24947beb99351da46a5aef9
1029 08:52:33.758995 (+ 3640us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.759094 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6124324}
I20251029 08:52:33.764763 1974 tablet_replica.cc:333] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.765017 1974 raft_consensus.cc:2243] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 08:52:33.765036 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 21d409c2e24947beb99351da46a5aef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d409c2e24947beb99351da46a5aef9 already in progress: creating tablet
I20251029 08:52:33.765661 1176 ts_tablet_manager.cc:1929] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.765787 1176 log.cc:1199] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/49d9c29b93a643ec837169a20c59de33
I20251029 08:52:33.766083 1176 ts_tablet_manager.cc:1950] T 49d9c29b93a643ec837169a20c59de33 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.766412 1974 raft_consensus.cc:2272] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.766993 1974 ts_tablet_manager.cc:1916] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.767077 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 49d9c29b93a643ec837169a20c59de33 (table default.Table_d1c5d76d3bd24eca923ed08a72bbc56b [id=af35c0b34122466c9e057ffde58e8836]) successfully deleted
I20251029 08:52:33.770305 1176 tablet_replica.cc:333] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.770502 1176 raft_consensus.cc:2243] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.770663 1176 raft_consensus.cc:2272] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.772532 1176 ts_tablet_manager.cc:1916] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.779882 2896 raft_consensus.cc:493] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.780154 1178 tablet_bootstrap.cc:492] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.780141 2896 raft_consensus.cc:515] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.780278 1178 ts_tablet_manager.cc:1403] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.000s
I20251029 08:52:33.780875 2896 leader_election.cc:290] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:33.780992 1178 raft_consensus.cc:359] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.781138 1178 raft_consensus.cc:385] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.781183 1178 raft_consensus.cc:740] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.781361 1178 consensus_queue.cc:260] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.781850 1178 ts_tablet_manager.cc:1434] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:33.785142 1176 ts_tablet_manager.cc:1929] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.785243 1176 log.cc:1199] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/65eaf12e45c54bd1b0aedc647dc90436
I20251029 08:52:33.785526 1176 ts_tablet_manager.cc:1950] T 65eaf12e45c54bd1b0aedc647dc90436 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.785934 1178 tablet_bootstrap.cc:492] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.787596 1178 tablet_bootstrap.cc:654] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.788290 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:33.788460 774 raft_consensus.cc:2468] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:33.790493 998 leader_election.cc:304] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
I20251029 08:52:33.790755 2359 consensus_queue.cc:1048] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 08:52:33.791170 2359 raft_consensus.cc:2804] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:33.791208 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 65eaf12e45c54bd1b0aedc647dc90436 (table default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]) successfully deleted
I20251029 08:52:33.791257 2359 raft_consensus.cc:493] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:33.791333 2359 raft_consensus.cc:3060] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:33.791509 1176 tablet_replica.cc:333] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.791676 1176 raft_consensus.cc:2243] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.791805 1176 raft_consensus.cc:2272] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.792676 1176 ts_tablet_manager.cc:1916] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.794284 2359 raft_consensus.cc:515] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.779110 1177 tablet_bootstrap.cc:492] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.796568 1177 ts_tablet_manager.cc:1403] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.008s
I20251029 08:52:33.797255 1177 raft_consensus.cc:359] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.797410 1177 raft_consensus.cc:385] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.797466 1177 raft_consensus.cc:740] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.797657 1177 consensus_queue.cc:260] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:33.798151 1177 ts_tablet_manager.cc:1434] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 08:52:33.798410 1177 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.799960 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
I20251029 08:52:33.782250 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:33.800518 1177 tablet_bootstrap.cc:654] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:33.800498 996 leader_election.cc:343] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:33.800725 996 leader_election.cc:343] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.804673 1183 tablet_bootstrap.cc:492] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.804823 1183 ts_tablet_manager.cc:1403] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20251029 08:52:33.806372 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32679cbb1884f09addb0f7c26ac941e" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:33.806560 778 raft_consensus.cc:3060] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:33.806962 921 raft_consensus.cc:3060] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:33.808146 2359 leader_election.cc:290] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:33.809257 778 raft_consensus.cc:2468] T c32679cbb1884f09addb0f7c26ac941e P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:33.809517 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 9429f6abb50c4f66b2e1b565f303962d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.809693 998 leader_election.cc:304] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.810127 2359 raft_consensus.cc:2804] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:33.810438 2359 raft_consensus.cc:697] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:33.810796 2359 consensus_queue.cc:237] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.814245 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 15dae38bcced496e90ca8eec280a8317 (table default.Table_1c14de238c434c4084a2b70bdf36d9a1 [id=22cd3b05e2c74671a965cd770dcee5a9]). Expected version 1 got 0
I20251029 08:52:33.814386 1183 raft_consensus.cc:359] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.814574 1183 raft_consensus.cc:385] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.814664 1183 raft_consensus.cc:740] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.815928 1183 consensus_queue.cc:260] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.816946 588 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 47470a907c1949989f79e8a2b803dc25 (table default.Table_afe756d752d549e6835f0f24b39e3398 [id=bc8a939fa4ab47049d9b2b82b4acdb23]). Expected version 1 got 0
I20251029 08:52:33.817013 1183 ts_tablet_manager.cc:1434] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251029 08:52:33.817307 1183 tablet_bootstrap.cc:492] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.823563 759 tablet_service.cc:1505] Processing CreateTablet for tablet 02cf976c0f164a708ca80ed653319853 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.823968 2944 tablet.cc:1722] T ce4a3503830c481797133028f5f123d9 P 927689e0dc5e4bc79d4a9d79a873f088: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 08:52:33.824283 1183 tablet_bootstrap.cc:654] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.825076 903 tablet_service.cc:1505] Processing CreateTablet for tablet 02cf976c0f164a708ca80ed653319853 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.825649 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 02cf976c0f164a708ca80ed653319853 (DEFAULT_TABLE table=default.Table_be08b1df1890403fbf34357b83147e36 [id=80595802d76f491e96c295f7450f30e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:33.826076 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cf976c0f164a708ca80ed653319853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.828727 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cf976c0f164a708ca80ed653319853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.831794 1178 tablet_bootstrap.cc:492] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.831940 1178 ts_tablet_manager.cc:1403] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20251029 08:52:33.832396 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cf976c0f164a708ca80ed653319853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:33.832756 1178 raft_consensus.cc:359] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.832924 1178 raft_consensus.cc:385] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.832971 1178 raft_consensus.cc:740] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.833151 1178 consensus_queue.cc:260] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.833622 1178 ts_tablet_manager.cc:1434] T 9a678d92f6334b66989eb3f71d830257 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:33.833873 1178 tablet_bootstrap.cc:492] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.835405 1178 tablet_bootstrap.cc:654] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.840477 1176 ts_tablet_manager.cc:1929] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.840588 1176 log.cc:1199] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/d0d00a6b772d4034ac7fa764884c7081
I20251029 08:52:33.841729 1176 ts_tablet_manager.cc:1950] T d0d00a6b772d4034ac7fa764884c7081 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.842715 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet d0d00a6b772d4034ac7fa764884c7081 (table default.Table_d29aed5fe8f44c47a0f174f5d5fd9d5f [id=d20448292e8c40109ee4c584de13ea95]) successfully deleted
I20251029 08:52:33.847497 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 with delete_type TABLET_DATA_DELETED (replaced by tablet c21cfbfbde5541a8976aa9a9b50e8867) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.853794 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet ce4a3503830c481797133028f5f123d9 (table default.Table_a5ebbce3f0ef404699dae9a0c1c670c3 [id=e1da82f219d140ad8ee2c8d2786bd40e]). Expected version 2 got 0
I20251029 08:52:33.858937 1183 tablet_bootstrap.cc:492] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.859097 1183 ts_tablet_manager.cc:1403] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20251029 08:52:33.861115 1176 tablet_replica.cc:333] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.861343 1176 raft_consensus.cc:2243] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.861517 1176 raft_consensus.cc:2272] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.863947 1974 ts_tablet_manager.cc:1929] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.865242 1176 ts_tablet_manager.cc:1916] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.868424 588 master_service.cc:954] ReplaceTablet: received request to replace tablet 00829c7fa0fe4e379a1a3044550e5941 from {username='slave'} at 127.0.0.1:39506
I20251029 08:52:33.873819 1974 log.cc:1199] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/913fb69271e14fe3a8d0b29d0db0dc8e
I20251029 08:52:33.874321 1974 ts_tablet_manager.cc:1950] T 913fb69271e14fe3a8d0b29d0db0dc8e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.880362 588 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8395ff5ec92c4bbda9bc6641a9fbbb6b: Sending DeleteTablet for 3 replicas of tablet 00829c7fa0fe4e379a1a3044550e5941
I20251029 08:52:33.881198 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 with delete_type TABLET_DATA_DELETED (replaced by tablet 23aa4288b74441168088245c52915764) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.886793 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 with delete_type TABLET_DATA_DELETED (replaced by tablet 23aa4288b74441168088245c52915764) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.887248 588 catalog_manager.cc:6719] ReplaceTablet: tablet 00829c7fa0fe4e379a1a3044550e5941 deleted and replaced by tablet 23aa4288b74441168088245c52915764
I20251029 08:52:33.890015 1176 ts_tablet_manager.cc:1929] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.890126 1176 log.cc:1199] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b8f1283f5a9245a599e5576a1e84b4a5
I20251029 08:52:33.890412 1176 ts_tablet_manager.cc:1950] T b8f1283f5a9245a599e5576a1e84b4a5 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.891496 1176 tablet_replica.cc:333] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.891687 1176 raft_consensus.cc:2243] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.891844 1176 raft_consensus.cc:2272] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.893911 1183 raft_consensus.cc:359] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.894112 1183 raft_consensus.cc:385] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.894165 1183 raft_consensus.cc:740] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.894394 1183 consensus_queue.cc:260] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.895006 1183 ts_tablet_manager.cc:1434] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251029 08:52:33.895035 1176 ts_tablet_manager.cc:1916] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.895328 1183 tablet_bootstrap.cc:492] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.896178 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1140) took 6253 ms. Trace:
I20251029 08:52:33.897030 1177 tablet_bootstrap.cc:492] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.898859 1177 ts_tablet_manager.cc:1403] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.009s
I20251029 08:52:33.899546 1177 raft_consensus.cc:359] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.899734 1177 raft_consensus.cc:385] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.899818 1177 raft_consensus.cc:740] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.898872 1974 rpcz_store.cc:276] 1029 08:52:27.642435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.642585 (+ 150us) service_pool.cc:225] Handling call
1029 08:52:33.764710 (+6122125us) ts_tablet_manager.cc:1151] Deleting tablet 913fb69271e14fe3a8d0b29d0db0dc8e
1029 08:52:33.764720 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.833082 (+ 68362us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.863808 (+ 30726us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.896130 (+ 32322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6121823}
I20251029 08:52:33.900477 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 913fb69271e14fe3a8d0b29d0db0dc8e (table default.Table_43572296e299444e8456983d5b1982f3 [id=f62019a2d7704e879a124daa2ab81268]) successfully deleted
I20251029 08:52:33.901468 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1144) took 6221 ms. Trace:
I20251029 08:52:33.901566 1974 rpcz_store.cc:276] 1029 08:52:27.680481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.680535 (+ 54us) service_pool.cc:225] Handling call
1029 08:52:33.901314 (+6220779us) ts_tablet_manager.cc:1151] Deleting tablet 1fb9c8c7286149128c2741635c242341
1029 08:52:33.901326 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.901425 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6220552}
W20251029 08:52:33.902432 996 consensus_peers.cc:597] T 471737aa35e94dcfaa2b3451cc8f2014 P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 08:52:33.902607 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 1fb9c8c7286149128c2741635c242341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb9c8c7286149128c2741635c242341 already in progress: creating tablet
I20251029 08:52:33.902768 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b8f1283f5a9245a599e5576a1e84b4a5 (table default.Table_183dac6abe3b4b6d8ef9614f941af3a0 [id=a99e94132c4e414ab2722fcb747ad901]) successfully deleted
I20251029 08:52:33.902921 1974 tablet_replica.cc:333] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
W20251029 08:52:33.903004 584 catalog_manager.cc:4965] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): delete failed for tablet 7c5525b69c8a4450b6d6571d39abb079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c5525b69c8a4450b6d6571d39abb079 already in progress: creating tablet
I20251029 08:52:33.903113 1974 raft_consensus.cc:2243] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.903755 1974 raft_consensus.cc:2272] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.904052 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 15dae38bcced496e90ca8eec280a8317 (table default.Table_1c14de238c434c4084a2b70bdf36d9a1 [id=22cd3b05e2c74671a965cd770dcee5a9]). Expected version 1 got 0
I20251029 08:52:33.904376 595 catalog_manager.cc:5649] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:33.904695 1974 ts_tablet_manager.cc:1916] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.906723 1183 tablet_bootstrap.cc:654] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.903966 1177 consensus_queue.cc:260] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.907290 1177 ts_tablet_manager.cc:1434] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 08:52:33.908306 1177 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.909185 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet c32679cbb1884f09addb0f7c26ac941e (table default.Table_062158d11a604fa6b931f3c1501da486 [id=a19e19f35785473e8533055ad792e0fa]). Expected version 2 got 0
I20251029 08:52:33.909950 1177 tablet_bootstrap.cc:654] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.917899 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 65b20d53d6214bdc81ca205dbf922d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:19 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.921306 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 with delete_type TABLET_DATA_DELETED (replaced by tablet 23aa4288b74441168088245c52915764) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.922207 1178 tablet_bootstrap.cc:492] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.922345 1178 ts_tablet_manager.cc:1403] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.001s
I20251029 08:52:33.923141 1178 raft_consensus.cc:359] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.923313 1178 raft_consensus.cc:385] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.923370 1178 raft_consensus.cc:740] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.923573 1178 consensus_queue.cc:260] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.924078 1178 ts_tablet_manager.cc:1434] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 08:52:33.924810 1178 tablet_bootstrap.cc:492] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.930632 1176 ts_tablet_manager.cc:1929] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:33.930749 1176 log.cc:1199] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/38032950744b4426927157fad0962dfb
I20251029 08:52:33.932161 1178 tablet_bootstrap.cc:654] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.934345 759 tablet_service.cc:1552] Processing DeleteTablet for tablet a712185fe4aa4f8faf1ce2df229c11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.937825 1178 tablet_bootstrap.cc:492] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.937968 1178 ts_tablet_manager.cc:1403] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20251029 08:52:33.938731 1178 raft_consensus.cc:359] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.938922 1178 raft_consensus.cc:385] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.938982 1178 raft_consensus.cc:740] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.939174 1178 consensus_queue.cc:260] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.939648 1178 ts_tablet_manager.cc:1434] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:33.939935 1178 tablet_bootstrap.cc:492] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:33.940140 1974 ts_tablet_manager.cc:1929] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 08:52:33.940351 1974 log.cc:1199] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/0cc95257c937454aa6cd55b5a9394a56
I20251029 08:52:33.941432 1974 ts_tablet_manager.cc:1950] T 0cc95257c937454aa6cd55b5a9394a56 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.942517 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1142) took 6279 ms. Trace:
I20251029 08:52:33.943490 1974 rpcz_store.cc:276] 1029 08:52:27.662985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.689948 (+ 26963us) service_pool.cc:225] Handling call
1029 08:52:33.902848 (+6212900us) ts_tablet_manager.cc:1151] Deleting tablet 0cc95257c937454aa6cd55b5a9394a56
1029 08:52:33.902857 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.917502 (+ 14645us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.934529 (+ 17027us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.942467 (+ 7938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6212730}
I20251029 08:52:33.944144 1176 ts_tablet_manager.cc:1950] T 38032950744b4426927157fad0962dfb P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:33.944870 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 0cc95257c937454aa6cd55b5a9394a56 (table default.Table_f096ddf6ddeb4b64b780d57471d56d8d [id=13d97eefb49f42cc94cb040ed86f4aab]) successfully deleted
I20251029 08:52:33.945302 1974 tablet_replica.cc:333] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.945937 1974 raft_consensus.cc:2243] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.946599 1974 raft_consensus.cc:2272] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.948477 1974 ts_tablet_manager.cc:1916] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.952207 1178 tablet_bootstrap.cc:654] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:33.953724 2972 raft_consensus.cc:493] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.953946 2972 raft_consensus.cc:515] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.954823 2972 leader_election.cc:290] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:33.955966 861 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 7a355c68ecfb45ff8c396873d91268a8
W20251029 08:52:33.956831 861 leader_election.cc:343] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7a355c68ecfb45ff8c396873d91268a8
I20251029 08:52:33.956956 861 leader_election.cc:304] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:33.957813 2972 raft_consensus.cc:2749] T 7a355c68ecfb45ff8c396873d91268a8 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.962515 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 38032950744b4426927157fad0962dfb (table default.Table_b60ce497df09480daa025d919c6509e3 [id=4c0bf44a89564ada810be44e6fe2cd95]) successfully deleted
I20251029 08:52:33.962867 1176 tablet_replica.cc:333] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:33.963071 1176 raft_consensus.cc:2243] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:33.963215 1176 raft_consensus.cc:2272] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.964180 1178 tablet_bootstrap.cc:492] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:33.964315 1178 ts_tablet_manager.cc:1403] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20251029 08:52:33.965065 1178 raft_consensus.cc:359] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.965229 1178 raft_consensus.cc:385] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.965282 1178 raft_consensus.cc:740] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.965468 1178 consensus_queue.cc:260] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.968284 759 tablet_service.cc:1552] Processing DeleteTablet for tablet b7a6008b4e6243ccaf166a31f1a9f62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:21 UTC) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:33.968811 1178 ts_tablet_manager.cc:1434] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:33.970785 1974 ts_tablet_manager.cc:1929] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.971552 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 485c9946f3f14a2aab11f4cdaa70a5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:20 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.971652 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 58932f38c0d64aec9a02238eaea56a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:33.972162 1974 log.cc:1199] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/19fdc381a0d949b0b93cef399daa3021
I20251029 08:52:33.972776 1974 ts_tablet_manager.cc:1950] T 19fdc381a0d949b0b93cef399daa3021 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.974223 1176 ts_tablet_manager.cc:1916] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.976852 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1141) took 6327 ms. Trace:
I20251029 08:52:33.977017 1974 rpcz_store.cc:276] 1029 08:52:27.649223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.691409 (+ 42186us) service_pool.cc:225] Handling call
1029 08:52:33.945234 (+6253825us) ts_tablet_manager.cc:1151] Deleting tablet 19fdc381a0d949b0b93cef399daa3021
1029 08:52:33.945244 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.957044 (+ 11800us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.970605 (+ 13561us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.976810 (+ 6205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6253667}
I20251029 08:52:33.977588 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1150) took 6147 ms. Trace:
I20251029 08:52:33.977627 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 19fdc381a0d949b0b93cef399daa3021 (table default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]) successfully deleted
I20251029 08:52:33.977764 1974 rpcz_store.cc:276] 1029 08:52:27.830317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:27.830366 (+ 49us) service_pool.cc:225] Handling call
1029 08:52:33.977428 (+6147062us) ts_tablet_manager.cc:1151] Deleting tablet 980e7125f59140c8876e4befc1becd78
1029 08:52:33.977440 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.977553 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6146880}
I20251029 08:52:33.978122 1974 tablet_replica.cc:333] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.978341 1974 raft_consensus.cc:2243] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
W20251029 08:52:33.978451 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 980e7125f59140c8876e4befc1becd78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 980e7125f59140c8876e4befc1becd78 already in progress: creating tablet
I20251029 08:52:33.978756 1974 raft_consensus.cc:2272] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:33.979387 1974 ts_tablet_manager.cc:1916] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:33.979609 1177 tablet_bootstrap.cc:492] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:33.979750 1177 ts_tablet_manager.cc:1403] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.006s
I20251029 08:52:33.980695 2920 raft_consensus.cc:493] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:33.980830 2920 raft_consensus.cc:515] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.981410 2920 leader_election.cc:290] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:33.980466 1177 raft_consensus.cc:359] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.981663 1177 raft_consensus.cc:385] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.981752 1177 raft_consensus.cc:740] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.981841 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2941ac88a7d453bb36f311c363231ab" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:33.981999 1177 consensus_queue.cc:260] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.982498 1177 ts_tablet_manager.cc:1434] T 9907d044daa145168417371c2bd66544 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251029 08:52:33.982721 720 leader_election.cc:343] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.982735 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2941ac88a7d453bb36f311c363231ab" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:33.983227 718 leader_election.cc:343] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:33.983322 718 leader_election.cc:304] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:33.983980 2920 raft_consensus.cc:2749] T d2941ac88a7d453bb36f311c363231ab P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:33.985023 1177 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:33.985960 1183 tablet_bootstrap.cc:492] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:33.986111 1183 ts_tablet_manager.cc:1403] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.008s
I20251029 08:52:33.986791 1183 raft_consensus.cc:359] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.986959 1183 raft_consensus.cc:385] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:33.987017 1183 raft_consensus.cc:740] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:33.987247 1183 consensus_queue.cc:260] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:33.988951 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 9907d044daa145168417371c2bd66544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:10 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:33.991509 1974 ts_tablet_manager.cc:1929] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:33.991714 1974 log.cc:1199] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/4e213713757949caa3f504f70026b7b5
I20251029 08:52:33.992187 1974 ts_tablet_manager.cc:1950] T 4e213713757949caa3f504f70026b7b5 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:33.992820 1183 ts_tablet_manager.cc:1434] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:33.995303 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1158) took 5973 ms. Trace:
I20251029 08:52:33.995467 1974 rpcz_store.cc:276] 1029 08:52:28.021626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.021689 (+ 63us) service_pool.cc:225] Handling call
1029 08:52:33.978063 (+5956374us) ts_tablet_manager.cc:1151] Deleting tablet 4e213713757949caa3f504f70026b7b5
1029 08:52:33.978071 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.983253 (+ 5182us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.991371 (+ 8118us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:33.995253 (+ 3882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956158}
I20251029 08:52:33.996183 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1162) took 5787 ms. Trace:
I20251029 08:52:33.996359 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 4e213713757949caa3f504f70026b7b5 (table default.Table_4ab4c150f8dd4dfdba46c3555d68e653 [id=ebdb88ba8e354cbda8a3ff4e89dd05a5]) successfully deleted
I20251029 08:52:33.996335 1974 rpcz_store.cc:276] 1029 08:52:28.208282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.210031 (+ 1749us) service_pool.cc:225] Handling call
1029 08:52:33.995973 (+5785942us) ts_tablet_manager.cc:1151] Deleting tablet 56f9308235f145ee838a55434f772bfa
1029 08:52:33.995983 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.996149 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5785724}
W20251029 08:52:33.996783 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 56f9308235f145ee838a55434f772bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f9308235f145ee838a55434f772bfa already in progress: creating tablet
I20251029 08:52:33.997143 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1164) took 5747 ms. Trace:
I20251029 08:52:33.997292 1974 rpcz_store.cc:276] 1029 08:52:28.249984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.250030 (+ 46us) service_pool.cc:225] Handling call
1029 08:52:33.997019 (+5746989us) ts_tablet_manager.cc:1151] Deleting tablet 420c3357ebff4dd08df4ed2477b40ddf
1029 08:52:33.997028 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.997099 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5746774}
I20251029 08:52:33.997557 1183 tablet_bootstrap.cc:492] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:33.997711 1177 tablet_bootstrap.cc:654] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:33.997792 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 420c3357ebff4dd08df4ed2477b40ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420c3357ebff4dd08df4ed2477b40ddf already in progress: creating tablet
I20251029 08:52:33.998195 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1163) took 5748 ms. Trace:
I20251029 08:52:33.998333 1974 rpcz_store.cc:276] 1029 08:52:28.249848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.250343 (+ 495us) service_pool.cc:225] Handling call
1029 08:52:33.998062 (+5747719us) ts_tablet_manager.cc:1151] Deleting tablet bdfda0866a614165824d1a3f7ad89af7
1029 08:52:33.998076 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:33.998158 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5747563}
W20251029 08:52:33.998808 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet bdfda0866a614165824d1a3f7ad89af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdfda0866a614165824d1a3f7ad89af7 already in progress: creating tablet
I20251029 08:52:33.999159 1974 tablet_replica.cc:333] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:33.999397 1974 raft_consensus.cc:2243] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:33.999835 1974 raft_consensus.cc:2272] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.000339 1178 tablet_bootstrap.cc:492] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:34.002056 1178 tablet_bootstrap.cc:654] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.004181 3005 raft_consensus.cc:493] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.004323 3005 raft_consensus.cc:515] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.004962 3005 leader_election.cc:290] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.005220 1183 tablet_bootstrap.cc:654] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.008759 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.008749 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:34.008955 778 raft_consensus.cc:2468] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:34.009380 998 leader_election.cc:304] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters:
W20251029 08:52:34.009528 996 leader_election.cc:343] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.009881 3005 raft_consensus.cc:2804] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:34.009969 3005 raft_consensus.cc:493] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:34.010046 3005 raft_consensus.cc:3060] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:34.010897 1974 ts_tablet_manager.cc:1916] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.012785 3005 raft_consensus.cc:515] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.013538 3005 leader_election.cc:290] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.013935 1177 tablet_bootstrap.cc:492] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.014169 1177 ts_tablet_manager.cc:1403] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.002s
I20251029 08:52:34.014338 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:34.014926 996 leader_election.cc:343] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.015600 1177 raft_consensus.cc:359] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.015851 1177 raft_consensus.cc:385] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.015966 1177 raft_consensus.cc:740] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.016237 1177 consensus_queue.cc:260] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.016588 778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15dae38bcced496e90ca8eec280a8317" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:34.016762 778 raft_consensus.cc:3060] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:34.023216 1177 ts_tablet_manager.cc:1434] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251029 08:52:34.025615 1177 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.026113 1176 ts_tablet_manager.cc:1929] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.027841 3003 raft_consensus.cc:493] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.027995 1177 tablet_bootstrap.cc:654] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.027979 3003 raft_consensus.cc:515] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.028664 3003 leader_election.cc:290] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.028915 1176 log.cc:1199] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/8017e8fe5f634ef0bf7f8be7c8aea033
I20251029 08:52:34.029266 1176 ts_tablet_manager.cc:1950] T 8017e8fe5f634ef0bf7f8be7c8aea033 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:34.030592 2938 ts_tablet_manager.cc:1929] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.031733 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485c9946f3f14a2aab11f4cdaa70a5b6" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.031931 1178 tablet_bootstrap.cc:492] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:34.032084 1178 ts_tablet_manager.cc:1403] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.004s
I20251029 08:52:34.032829 1178 raft_consensus.cc:359] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.032995 1178 raft_consensus.cc:385] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.033046 1178 raft_consensus.cc:740] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.033263 778 raft_consensus.cc:2468] T 15dae38bcced496e90ca8eec280a8317 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:34.033277 1178 consensus_queue.cc:260] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.033946 1178 ts_tablet_manager.cc:1434] T 633ace1413464e939e4e47782370e281 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:34.037417 1974 ts_tablet_manager.cc:1929] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.037609 1974 log.cc:1199] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d3955ff79faf42bb82f764aae79b85d9
I20251029 08:52:34.038434 1974 ts_tablet_manager.cc:1950] T d3955ff79faf42bb82f764aae79b85d9 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.039280 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1165) took 5787 ms. Trace:
I20251029 08:52:34.039469 1974 rpcz_store.cc:276] 1029 08:52:28.251727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.251780 (+ 53us) service_pool.cc:225] Handling call
1029 08:52:33.999092 (+5747312us) ts_tablet_manager.cc:1151] Deleting tablet d3955ff79faf42bb82f764aae79b85d9
1029 08:52:33.999101 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.034083 (+ 34982us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.037294 (+ 3211us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.039238 (+ 1944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5747126}
I20251029 08:52:34.039973 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d3955ff79faf42bb82f764aae79b85d9 (table default.Table_8360e68864e74c0b8acca762339062b4 [id=fbba296113094a2389acc968c256c1b9]) successfully deleted
I20251029 08:52:34.040518 1974 tablet_replica.cc:333] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.040866 1974 raft_consensus.cc:2243] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:34.041842 1974 raft_consensus.cc:2272] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.042696 1974 ts_tablet_manager.cc:1916] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.044157 1177 tablet_bootstrap.cc:492] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.044314 1177 ts_tablet_manager.cc:1403] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20251029 08:52:34.045053 1177 raft_consensus.cc:359] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.045233 1177 raft_consensus.cc:385] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.045311 1177 raft_consensus.cc:740] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.045539 1177 consensus_queue.cc:260] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.048363 1177 ts_tablet_manager.cc:1434] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 08:52:34.048668 1177 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.050499 1177 tablet_bootstrap.cc:654] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
W20251029 08:52:34.031178 998 leader_election.cc:343] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 485c9946f3f14a2aab11f4cdaa70a5b6
I20251029 08:52:34.051590 2938 log.cc:1199] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/a506e35afe6a4c87901f27c434f3b348
I20251029 08:52:34.051816 998 leader_election.cc:304] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.052008 2938 ts_tablet_manager.cc:1950] T a506e35afe6a4c87901f27c434f3b348 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.052546 3003 raft_consensus.cc:2804] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:34.052666 3003 raft_consensus.cc:697] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:34.052884 3003 consensus_queue.cc:237] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:34.054060 996 leader_election.cc:343] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.032631 1183 tablet_bootstrap.cc:492] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.054279 996 leader_election.cc:304] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.054462 1183 ts_tablet_manager.cc:1403] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20251029 08:52:34.054641 2896 raft_consensus.cc:2749] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.055172 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet a506e35afe6a4c87901f27c434f3b348 (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281]) successfully deleted
I20251029 08:52:34.055297 1183 raft_consensus.cc:359] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.055454 1183 raft_consensus.cc:385] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.055506 1183 raft_consensus.cc:740] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.054273 2938 tablet_replica.cc:333] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.055677 1183 consensus_queue.cc:260] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.055907 2938 raft_consensus.cc:2243] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.061134 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 57159c264d6641afa3415d33c1d32efe (table default.Table_80492c8792f54daaa09fb96f4fa0b0f5 [id=7a5c0a7295414daa823ea0df7d47764d]). Expected version 1 got 0
I20251029 08:52:34.061705 1183 ts_tablet_manager.cc:1434] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 08:52:34.062023 1183 tablet_bootstrap.cc:492] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.062716 2938 raft_consensus.cc:2272] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.063436 1183 tablet_bootstrap.cc:654] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.063592 1974 ts_tablet_manager.cc:1929] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.063783 1974 log.cc:1199] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/c9bbf6401423411089c9070f3b03f1a7
I20251029 08:52:34.064213 1974 ts_tablet_manager.cc:1950] T c9bbf6401423411089c9070f3b03f1a7 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.065026 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1166) took 5812 ms. Trace:
I20251029 08:52:34.065194 1974 rpcz_store.cc:276] 1029 08:52:28.252483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.252530 (+ 47us) service_pool.cc:225] Handling call
1029 08:52:34.040415 (+5787885us) ts_tablet_manager.cc:1151] Deleting tablet c9bbf6401423411089c9070f3b03f1a7
1029 08:52:34.040431 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.060058 (+ 19627us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.063467 (+ 3409us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.064982 (+ 1515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5787726}
I20251029 08:52:34.065881 2938 ts_tablet_manager.cc:1916] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.065481 903 tablet_service.cc:1505] Processing CreateTablet for tablet 23aa4288b74441168088245c52915764 (DEFAULT_TABLE table=default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:34.066226 1974 tablet_replica.cc:333] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.066303 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23aa4288b74441168088245c52915764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:34.066303 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet c9bbf6401423411089c9070f3b03f1a7 (table default.Table_18c407e625464254aa01f6cc9e12144f [id=58aabf0c50d64caa918b4d97574d7c5f]) successfully deleted
I20251029 08:52:34.066568 1974 raft_consensus.cc:2243] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.072409 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 23aa4288b74441168088245c52915764 (DEFAULT_TABLE table=default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:34.072688 1974 raft_consensus.cc:2272] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.072829 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23aa4288b74441168088245c52915764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:34.073276 1974 ts_tablet_manager.cc:1916] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.073609 759 tablet_service.cc:1505] Processing CreateTablet for tablet 23aa4288b74441168088245c52915764 (DEFAULT_TABLE table=default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 08:52:34.075776 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 57159c264d6641afa3415d33c1d32efe (table default.Table_80492c8792f54daaa09fb96f4fa0b0f5 [id=7a5c0a7295414daa823ea0df7d47764d]). Expected version 1 got 0
I20251029 08:52:34.077941 2920 raft_consensus.cc:493] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.078078 2920 raft_consensus.cc:515] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.078691 2920 leader_election.cc:290] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.079174 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8df243ce9eba41b7a050706733d7f441" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.079280 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8df243ce9eba41b7a050706733d7f441" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:34.079434 1974 ts_tablet_manager.cc:1929] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.079547 1974 log.cc:1199] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d50e2fdfc0e64516a41c90f5a56f4c89
W20251029 08:52:34.079813 718 leader_election.cc:343] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.075999 595 catalog_manager.cc:5649] T 15dae38bcced496e90ca8eec280a8317 P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } }
I20251029 08:52:34.081298 595 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 15dae38bcced496e90ca8eec280a8317 (table default.Table_1c14de238c434c4084a2b70bdf36d9a1 [id=22cd3b05e2c74671a965cd770dcee5a9]). Expected version 1 got 0
W20251029 08:52:34.082170 720 leader_election.cc:343] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.082286 720 leader_election.cc:304] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.082230 2938 ts_tablet_manager.cc:1929] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.082495 2938 log.cc:1199] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/34729c3f97a44713a352bde297a80027
I20251029 08:52:34.082679 2920 raft_consensus.cc:2749] T 8df243ce9eba41b7a050706733d7f441 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.082919 2938 ts_tablet_manager.cc:1950] T 34729c3f97a44713a352bde297a80027 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.083927 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 34729c3f97a44713a352bde297a80027 (table default.Table_15cd2138e80e429db3004528a89ec2bf [id=a8a2e644aa73491b9d22e81babf2e582]) successfully deleted
I20251029 08:52:34.084570 1974 ts_tablet_manager.cc:1950] T d50e2fdfc0e64516a41c90f5a56f4c89 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:34.085119 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 7c5525b69c8a4450b6d6571d39abb079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c5525b69c8a4450b6d6571d39abb079 already in progress: creating tablet
I20251029 08:52:34.086392 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1167) took 5833 ms. Trace:
I20251029 08:52:34.086673 2938 tablet_replica.cc:333] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.086918 2938 raft_consensus.cc:2243] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.087122 2938 raft_consensus.cc:2272] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.087836 1178 tablet_bootstrap.cc:492] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:34.088934 1176 tablet_replica.cc:333] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:34.089145 1176 raft_consensus.cc:2243] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.089280 1176 raft_consensus.cc:2272] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.089650 3003 raft_consensus.cc:493] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.089740 1178 tablet_bootstrap.cc:654] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.089811 1176 ts_tablet_manager.cc:1916] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.089771 3003 raft_consensus.cc:515] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.090682 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 8017e8fe5f634ef0bf7f8be7c8aea033 (table default.Table_b66efc855f5b4b1ea6f2110c19c3812b [id=033c1c3305724f0f9b37fd9729616d46]) successfully deleted
I20251029 08:52:34.091034 3003 leader_election.cc:290] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.092851 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23aa4288b74441168088245c52915764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 08:52:34.093490 416 master-stress-test.cc:579] Killed test thread 22/25
I20251029 08:52:34.093616 416 master-stress-test.cc:577] Killing test thread 23/25
I20251029 08:52:34.093711 416 master-stress-test.cc:579] Killed test thread 23/25
I20251029 08:52:34.093789 416 master-stress-test.cc:577] Killing test thread 24/25
I20251029 08:52:34.093865 416 master-stress-test.cc:579] Killed test thread 24/25
I20251029 08:52:34.093940 416 master-stress-test.cc:577] Killing test thread 25/25
I20251029 08:52:34.094010 416 master-stress-test.cc:579] Killed test thread 25/25
I20251029 08:52:34.094182 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskLKoUQR/build/asan/bin/kudu pid 695
I20251029 08:52:34.090749 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:34.095566 996 leader_election.cc:343] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.095805 1974 rpcz_store.cc:276] 1029 08:52:28.253133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.253175 (+ 42us) service_pool.cc:225] Handling call
1029 08:52:34.066153 (+5812978us) ts_tablet_manager.cc:1151] Deleting tablet d50e2fdfc0e64516a41c90f5a56f4c89
1029 08:52:34.066162 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.076407 (+ 10245us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.079309 (+ 2902us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.086337 (+ 7028us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5812822}
I20251029 08:52:34.096498 2938 ts_tablet_manager.cc:1916] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.097059 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d50e2fdfc0e64516a41c90f5a56f4c89 (table default.Table_064ffeda7eaa4da6854b9395b0c98345 [id=2c2d4b65043e4076968b58f0ebb69ca9]) successfully deleted
I20251029 08:52:34.097568 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1169) took 5812 ms. Trace:
I20251029 08:52:34.099035 1974 rpcz_store.cc:276] 1029 08:52:28.284873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.284930 (+ 57us) service_pool.cc:225] Handling call
1029 08:52:34.097418 (+5812488us) ts_tablet_manager.cc:1151] Deleting tablet 5e35d40b730a4a7883c324a6cc93b161
1029 08:52:34.097432 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.097530 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5812293}
W20251029 08:52:34.099820 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5e35d40b730a4a7883c324a6cc93b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e35d40b730a4a7883c324a6cc93b161 already in progress: creating tablet
I20251029 08:52:34.100461 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1168) took 5815 ms. Trace:
I20251029 08:52:34.101755 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 9499d0099d1c4edc843fb8d29ca3b1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.103695 1974 rpcz_store.cc:276] 1029 08:52:28.284725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.285316 (+ 591us) service_pool.cc:225] Handling call
1029 08:52:34.100284 (+5814968us) ts_tablet_manager.cc:1151] Deleting tablet c5d9527d294c432db2ccf60af84731b0
1029 08:52:34.100300 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.100420 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5814804}
W20251029 08:52:34.107136 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c5d9527d294c432db2ccf60af84731b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d9527d294c432db2ccf60af84731b0 already in progress: creating tablet
I20251029 08:52:34.107674 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1179) took 5584 ms. Trace:
I20251029 08:52:34.108222 1974 rpcz_store.cc:276] 1029 08:52:28.522951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:28.523820 (+ 869us) service_pool.cc:225] Handling call
1029 08:52:34.107521 (+5583701us) ts_tablet_manager.cc:1151] Deleting tablet e14d1d3421a542ec960d21453bb713ec
1029 08:52:34.107534 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.107638 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5583557}
W20251029 08:52:34.109012 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e14d1d3421a542ec960d21453bb713ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14d1d3421a542ec960d21453bb713ec already in progress: creating tablet
I20251029 08:52:34.109504 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1192) took 5073 ms. Trace:
I20251029 08:52:34.110141 1974 rpcz_store.cc:276] 1029 08:52:29.036356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.036681 (+ 325us) service_pool.cc:225] Handling call
1029 08:52:34.109333 (+5072652us) ts_tablet_manager.cc:1151] Deleting tablet d1d2df9e1e744ed1a6cf549ab57ed266
1029 08:52:34.109348 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.109456 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5072441}
W20251029 08:52:34.110749 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d1d2df9e1e744ed1a6cf549ab57ed266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d2df9e1e744ed1a6cf549ab57ed266 already in progress: creating tablet
I20251029 08:52:34.111198 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1193) took 4904 ms. Trace:
I20251029 08:52:34.111910 1974 rpcz_store.cc:276] 1029 08:52:29.206269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.216699 (+ 10430us) service_pool.cc:225] Handling call
1029 08:52:34.111066 (+4894367us) ts_tablet_manager.cc:1151] Deleting tablet c9b611ae08a247d3ba02eb177c6b1b12
1029 08:52:34.111075 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.111160 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4893951}
I20251029 08:52:34.113040 1183 tablet_bootstrap.cc:492] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.113186 1183 ts_tablet_manager.cc:1403] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
W20251029 08:52:34.113540 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet c9b611ae08a247d3ba02eb177c6b1b12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b611ae08a247d3ba02eb177c6b1b12 already in progress: creating tablet
I20251029 08:52:34.113903 1183 raft_consensus.cc:359] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.114089 1183 raft_consensus.cc:385] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.114158 1183 raft_consensus.cc:740] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.114398 1183 consensus_queue.cc:260] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.114964 1178 tablet_bootstrap.cc:492] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:34.115131 1178 ts_tablet_manager.cc:1403] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20251029 08:52:34.115989 1178 raft_consensus.cc:359] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.116245 1178 raft_consensus.cc:385] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.116334 1178 raft_consensus.cc:740] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.116597 1178 consensus_queue.cc:260] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.117375 1178 ts_tablet_manager.cc:1434] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 08:52:34.118279 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1194) took 4888 ms. Trace:
I20251029 08:52:34.118453 1974 rpcz_store.cc:276] 1029 08:52:29.225721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.277167 (+ 51446us) service_pool.cc:225] Handling call
1029 08:52:34.113906 (+4836739us) ts_tablet_manager.cc:1151] Deleting tablet b61829fcd59d4ad4ab427270b7089d57
1029 08:52:34.113919 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.114024 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4778220}
I20251029 08:52:34.121390 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 42de32d87ebb451090e0310e36e64277 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:22 UTC) from {username='slave'} at 127.0.0.1:37708
W20251029 08:52:34.122359 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b61829fcd59d4ad4ab427270b7089d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61829fcd59d4ad4ab427270b7089d57 already in progress: creating tablet
I20251029 08:52:34.122794 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1197) took 4445 ms. Trace:
I20251029 08:52:34.123240 1974 rpcz_store.cc:276] 1029 08:52:29.677641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.677876 (+ 235us) service_pool.cc:225] Handling call
1029 08:52:34.122668 (+4444792us) ts_tablet_manager.cc:1151] Deleting tablet 1dd62833c9d548adabbbf71856a4b181
1029 08:52:34.122678 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.122746 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4444614}
W20251029 08:52:34.123736 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 1dd62833c9d548adabbbf71856a4b181 because the tablet was not found. No further retry: Not found: Tablet not found: 1dd62833c9d548adabbbf71856a4b181
I20251029 08:52:34.123833 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6a8c25ba4e9c410fa6d12949d6e595d7 (table default.Table_642c013e1f074c5c983b2df65a41dcb3 [id=1e36c914f03b48eaa7061b7731ba512a]). Expected version 1 got 0
I20251029 08:52:34.124207 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1198) took 4446 ms. Trace:
I20251029 08:52:34.124922 1177 tablet_bootstrap.cc:492] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.125027 903 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bd490da0b0e47fc965343c123a27b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:17 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.125291 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f2b34642704cd6bb966b6dcd494d51 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f91eb858ae344549a78357e04f7e938) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.125286 1177 ts_tablet_manager.cc:1403] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.009s
I20251029 08:52:34.126266 1178 tablet_bootstrap.cc:492] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:34.126225 1177 raft_consensus.cc:359] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.126497 1177 raft_consensus.cc:385] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.126601 1177 raft_consensus.cc:740] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.126866 1177 consensus_queue.cc:260] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.128837 1176 ts_tablet_manager.cc:1929] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.128960 1176 log.cc:1199] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/00829c7fa0fe4e379a1a3044550e5941
I20251029 08:52:34.129243 1176 ts_tablet_manager.cc:1950] T 00829c7fa0fe4e379a1a3044550e5941 P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:34.130010 2972 raft_consensus.cc:493] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.130158 2972 raft_consensus.cc:515] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.130692 2972 leader_election.cc:290] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.131569 1177 ts_tablet_manager.cc:1434] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20251029 08:52:34.131660 861 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 65b20d53d6214bdc81ca205dbf922d7b
I20251029 08:52:34.131904 1177 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
W20251029 08:52:34.132272 861 leader_election.cc:343] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 65b20d53d6214bdc81ca205dbf922d7b
I20251029 08:52:34.132324 1183 ts_tablet_manager.cc:1434] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251029 08:52:34.132402 861 leader_election.cc:304] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.132838 2972 raft_consensus.cc:2749] T 65b20d53d6214bdc81ca205dbf922d7b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.135018 1183 tablet_bootstrap.cc:492] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.136257 1177 tablet_bootstrap.cc:654] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.136402 1183 tablet_bootstrap.cc:654] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.146371 1178 tablet_bootstrap.cc:654] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.147866 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 7046bb1fcdd34d308aecddfdf0752911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.124357 1974 rpcz_store.cc:276] 1029 08:52:29.677739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.678083 (+ 344us) service_pool.cc:225] Handling call
1029 08:52:34.124073 (+4445990us) ts_tablet_manager.cc:1151] Deleting tablet aa411391c63146e7acb8b4c6edbe12a0
1029 08:52:34.124083 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.124149 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4445848}
W20251029 08:52:34.148661 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet aa411391c63146e7acb8b4c6edbe12a0 because the tablet was not found. No further retry: Not found: Tablet not found: aa411391c63146e7acb8b4c6edbe12a0
I20251029 08:52:34.148800 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" is_pre_election: true
I20251029 08:52:34.149006 774 raft_consensus.cc:2468] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 0.
I20251029 08:52:34.149168 1974 tablet_replica.cc:333] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.149981 998 leader_election.cc:304] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.150395 3003 raft_consensus.cc:2804] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 08:52:34.150467 3003 raft_consensus.cc:493] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 08:52:34.150537 3003 raft_consensus.cc:3060] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:34.151005 1974 raft_consensus.cc:2243] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:34.151392 1974 pending_rounds.cc:70] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:34.151460 1974 pending_rounds.cc:77] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037680682172416 op_type: NO_OP noop_request { }
I20251029 08:52:34.151566 1974 raft_consensus.cc:2889] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:34.151660 1974 raft_consensus.cc:2272] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.152272 1974 ts_tablet_manager.cc:1916] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.153255 3003 raft_consensus.cc:515] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.153864 3003 leader_election.cc:290] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Requested vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.155279 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088"
W20251029 08:52:34.155750 996 leader_election.cc:343] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.155864 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57159c264d6641afa3415d33c1d32efe" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0"
I20251029 08:52:34.156180 774 raft_consensus.cc:3060] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 08:52:34.159015 2938 ts_tablet_manager.cc:1929] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.159535 774 raft_consensus.cc:2468] T 57159c264d6641afa3415d33c1d32efe P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730d0f5b7abf43a293b7df13d01ce4ee in term 1.
I20251029 08:52:34.161250 1177 tablet_bootstrap.cc:492] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.161471 1177 ts_tablet_manager.cc:1403] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20251029 08:52:34.162361 2972 raft_consensus.cc:493] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 730d0f5b7abf43a293b7df13d01ce4ee)
I20251029 08:52:34.162535 2972 raft_consensus.cc:515] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } }
I20251029 08:52:34.163112 2972 leader_election.cc:290] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:34.164422 861 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: e1e2be77182e4259b76d20cae4e9e478
I20251029 08:52:34.162310 1177 raft_consensus.cc:359] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.165185 1177 raft_consensus.cc:385] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.165318 1177 raft_consensus.cc:740] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.165629 1177 consensus_queue.cc:260] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.166383 1177 ts_tablet_manager.cc:1434] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20251029 08:52:34.167057 861 leader_election.cc:343] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: e1e2be77182e4259b76d20cae4e9e478
I20251029 08:52:34.167198 861 leader_election.cc:304] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.167650 2972 raft_consensus.cc:2749] T e1e2be77182e4259b76d20cae4e9e478 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 08:52:34.168725 1177 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.170555 1177 tablet_bootstrap.cc:654] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.168119 998 leader_election.cc:304] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.172740 1178 tablet_bootstrap.cc:492] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:34.172845 3003 raft_consensus.cc:2804] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 FOLLOWER]: Leader election won for term 1
I20251029 08:52:34.172894 1178 ts_tablet_manager.cc:1403] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20251029 08:52:34.170894 1183 tablet_bootstrap.cc:492] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.173069 1183 ts_tablet_manager.cc:1403] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251029 08:52:34.173362 3003 raft_consensus.cc:697] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [term 1 LEADER]: Becoming Leader. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Running, Role: LEADER
I20251029 08:52:34.173643 3003 consensus_queue.cc:237] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.173755 1183 raft_consensus.cc:359] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.173877 1183 raft_consensus.cc:385] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.173930 1183 raft_consensus.cc:740] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.174042 1178 raft_consensus.cc:359] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.174096 1183 consensus_queue.cc:260] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.174252 1178 raft_consensus.cc:385] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.174319 1178 raft_consensus.cc:740] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.174516 1178 consensus_queue.cc:260] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.174665 1183 ts_tablet_manager.cc:1434] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:34.174966 1183 tablet_bootstrap.cc:492] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.175055 1178 ts_tablet_manager.cc:1434] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:34.176601 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 663d6587a5e14519afbb0f94d691d1b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ed4f048f1ae4d70aa1b88116a57e2bd) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.177137 1183 tablet_bootstrap.cc:654] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.177887 2938 log.cc:1199] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d5bb0fd4c62d47a2b10719ebd55851c0
I20251029 08:52:34.178306 2938 ts_tablet_manager.cc:1950] T d5bb0fd4c62d47a2b10719ebd55851c0 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.179378 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet d5bb0fd4c62d47a2b10719ebd55851c0 (table default.Table_11a78e5b37424289a184f8bccbd263d4 [id=c54d460a25104cc19f43a5517c203b31]) successfully deleted
W20251029 08:52:34.180595 584 catalog_manager.cc:4965] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): delete failed for tablet 00829c7fa0fe4e379a1a3044550e5941 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00829c7fa0fe4e379a1a3044550e5941 already in progress: creating tablet
I20251029 08:52:34.180641 1178 tablet_bootstrap.cc:492] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:34.182039 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet 00829c7fa0fe4e379a1a3044550e5941 (table default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]) successfully deleted
I20251029 08:52:34.182379 2938 tablet_replica.cc:333] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.182458 1178 tablet_bootstrap.cc:654] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.183127 2938 raft_consensus.cc:2243] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.188339 2938 raft_consensus.cc:2272] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.190104 1176 tablet_replica.cc:333] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:34.190402 1176 raft_consensus.cc:2243] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.190620 1176 raft_consensus.cc:2272] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.191217 1177 tablet_bootstrap.cc:492] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.191376 1177 ts_tablet_manager.cc:1403] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20251029 08:52:34.192123 2938 ts_tablet_manager.cc:1916] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.192975 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 87c98206d7fa409b8017abc72688db93 (table default.Table_0b522a4d5a9949f79a2a0923a519ec77 [id=5f1e417568044b43b5b50db23a0ecc2f]). Expected version 2 got 0
I20251029 08:52:34.193006 1183 tablet_bootstrap.cc:492] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.193168 1183 ts_tablet_manager.cc:1403] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20251029 08:52:34.193895 1183 raft_consensus.cc:359] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.194069 1183 raft_consensus.cc:385] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.194123 1183 raft_consensus.cc:740] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.194375 1183 consensus_queue.cc:260] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.195326 1176 ts_tablet_manager.cc:1916] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.195411 1183 ts_tablet_manager.cc:1434] T 9d4c4050fe044d04b76768d39ee7db7e P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 08:52:34.195709 1183 tablet_bootstrap.cc:492] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.197203 1183 tablet_bootstrap.cc:654] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.200260 1177 raft_consensus.cc:359] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.201246 1177 raft_consensus.cc:385] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.201380 1177 raft_consensus.cc:740] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.201709 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 6a8c25ba4e9c410fa6d12949d6e595d7 (table default.Table_642c013e1f074c5c983b2df65a41dcb3 [id=1e36c914f03b48eaa7061b7731ba512a]). Expected version 1 got 0
I20251029 08:52:34.206657 1177 consensus_queue.cc:260] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.207265 1177 ts_tablet_manager.cc:1434] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251029 08:52:34.207897 2938 ts_tablet_manager.cc:1929] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.208211 1178 tablet_bootstrap.cc:492] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0: No bootstrap required, opened a new log
I20251029 08:52:34.202904 1974 ts_tablet_manager.cc:1929] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.208410 1178 ts_tablet_manager.cc:1403] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20251029 08:52:34.208570 1974 log.cc:1199] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/3e65b3b1c8d14743aca1465d95ace95f
I20251029 08:52:34.209071 1974 ts_tablet_manager.cc:1950] T 3e65b3b1c8d14743aca1465d95ace95f P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.209196 1178 raft_consensus.cc:359] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.209393 1178 raft_consensus.cc:385] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.209476 1178 raft_consensus.cc:740] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8d80098fcb40ada903c3b2bacc2ea0, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.209723 1178 consensus_queue.cc:260] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.210254 1178 ts_tablet_manager.cc:1434] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:34.210611 1178 tablet_bootstrap.cc:492] T 6ec00cd64dc247da8442b6ba197f370a P 7a8d80098fcb40ada903c3b2bacc2ea0: Bootstrap starting.
I20251029 08:52:34.214169 1177 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.215135 594 catalog_manager.cc:5649] T 57159c264d6641afa3415d33c1d32efe P 730d0f5b7abf43a293b7df13d01ce4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3). New cstate: current_term: 1 leader_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20251029 08:52:34.215551 2961 raft_consensus.cc:493] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.215607 594 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 57159c264d6641afa3415d33c1d32efe (table default.Table_80492c8792f54daaa09fb96f4fa0b0f5 [id=7a5c0a7295414daa823ea0df7d47764d]). Expected version 1 got 0
I20251029 08:52:34.215713 2961 raft_consensus.cc:515] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.216159 1177 tablet_bootstrap.cc:654] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.216408 2961 leader_election.cc:290] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.210250 1183 tablet_bootstrap.cc:492] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.216635 1183 ts_tablet_manager.cc:1403] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 08:52:34.217381 1183 raft_consensus.cc:359] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.217536 1183 raft_consensus.cc:385] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.217599 1183 raft_consensus.cc:740] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.217841 1183 consensus_queue.cc:260] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.218454 1183 ts_tablet_manager.cc:1434] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:34.218739 1183 tablet_bootstrap.cc:492] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.219305 2938 log.cc:1199] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/485c9946f3f14a2aab11f4cdaa70a5b6
I20251029 08:52:34.222186 2938 ts_tablet_manager.cc:1950] T 485c9946f3f14a2aab11f4cdaa70a5b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.223412 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 485c9946f3f14a2aab11f4cdaa70a5b6 (table default.Table_5df6707d7a754de3b61f32cf68786b2e [id=2f663ee400904f1d9e21d15b83c58e24]) successfully deleted
I20251029 08:52:34.223855 2938 tablet_replica.cc:333] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.224509 2920 raft_consensus.cc:493] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.224689 2920 raft_consensus.cc:515] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.225345 2920 leader_election.cc:290] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.228410 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7a6008b4e6243ccaf166a31f1a9f62c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:34.229103 2938 raft_consensus.cc:2243] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.229391 2938 raft_consensus.cc:2272] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 08:52:34.229676 861 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 0bd490da0b0e47fc965343c123a27b22
W20251029 08:52:34.230302 720 leader_election.cc:343] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.232478 1178 tablet_bootstrap.cc:654] T 6ec00cd64dc247da8442b6ba197f370a P 7a8d80098fcb40ada903c3b2bacc2ea0: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.234687 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1199) took 4556 ms. Trace:
I20251029 08:52:34.234941 1974 rpcz_store.cc:276] 1029 08:52:29.677828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.678253 (+ 425us) service_pool.cc:225] Handling call
1029 08:52:34.149096 (+4470843us) ts_tablet_manager.cc:1151] Deleting tablet 3e65b3b1c8d14743aca1465d95ace95f
1029 08:52:34.149108 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.189893 (+ 40785us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.202767 (+ 12874us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.234631 (+ 31864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4470717}
I20251029 08:52:34.235520 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 3e65b3b1c8d14743aca1465d95ace95f (table default.Table_5820689813104d31a2a4a5bcf6d6c53a [id=1cb494fea7744742a882e34339b4d199]) successfully deleted
I20251029 08:52:34.236064 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1200) took 4552 ms. Trace:
I20251029 08:52:34.236327 1974 rpcz_store.cc:276] 1029 08:52:29.683294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.683576 (+ 282us) service_pool.cc:225] Handling call
1029 08:52:34.235887 (+4552311us) ts_tablet_manager.cc:1151] Deleting tablet 61b07eb4befa41f3a1b5f2e9ebc2277c
1029 08:52:34.235897 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.235975 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4552087}
I20251029 08:52:34.236980 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
W20251029 08:52:34.237048 582 catalog_manager.cc:4952] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 61b07eb4befa41f3a1b5f2e9ebc2277c because the tablet was not found. No further retry: Not found: Tablet not found: 61b07eb4befa41f3a1b5f2e9ebc2277c
I20251029 08:52:34.237519 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1202) took 4536 ms. Trace:
I20251029 08:52:34.237672 1974 rpcz_store.cc:276] 1029 08:52:29.701327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.701649 (+ 322us) service_pool.cc:225] Handling call
1029 08:52:34.237380 (+4535731us) ts_tablet_manager.cc:1151] Deleting tablet 297945c50119432a9f9d3e356d573879
1029 08:52:34.237390 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.237484 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535554}
W20251029 08:52:34.238116 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 297945c50119432a9f9d3e356d573879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 297945c50119432a9f9d3e356d573879 already in progress: creating tablet
I20251029 08:52:34.238848 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1203) took 4537 ms. Trace:
I20251029 08:52:34.238984 1974 rpcz_store.cc:276] 1029 08:52:29.701420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.702984 (+ 1564us) service_pool.cc:225] Handling call
1029 08:52:34.238700 (+4535716us) ts_tablet_manager.cc:1151] Deleting tablet adba5919a81348bb94569ca52fa4bded
1029 08:52:34.238709 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.238814 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535555}
W20251029 08:52:34.239416 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet adba5919a81348bb94569ca52fa4bded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adba5919a81348bb94569ca52fa4bded already in progress: creating tablet
I20251029 08:52:34.239689 2896 raft_consensus.cc:493] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.239845 2896 raft_consensus.cc:515] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.240581 2896 leader_election.cc:290] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.241377 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65eaf12e45c54bd1b0aedc647dc90436" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:34.241906 996 leader_election.cc:343] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.243755 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1204) took 4542 ms. Trace:
I20251029 08:52:34.243922 1974 rpcz_store.cc:276] 1029 08:52:29.701597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.703179 (+ 1582us) service_pool.cc:225] Handling call
1029 08:52:34.243604 (+4540425us) ts_tablet_manager.cc:1151] Deleting tablet 5d292c29d2b34086a5976da251f9ed5e
1029 08:52:34.243615 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.243714 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540294}
W20251029 08:52:34.244905 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 5d292c29d2b34086a5976da251f9ed5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d292c29d2b34086a5976da251f9ed5e already in progress: creating tablet
I20251029 08:52:34.245371 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1205) took 4541 ms. Trace:
I20251029 08:52:34.245507 1974 rpcz_store.cc:276] 1029 08:52:29.703570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.703614 (+ 44us) service_pool.cc:225] Handling call
1029 08:52:34.245240 (+4541626us) ts_tablet_manager.cc:1151] Deleting tablet 76dcdd90e4a54cbfb0cb62192c465c40
1029 08:52:34.245251 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.245335 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4541485}
W20251029 08:52:34.245985 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 76dcdd90e4a54cbfb0cb62192c465c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76dcdd90e4a54cbfb0cb62192c465c40 already in progress: creating tablet
I20251029 08:52:34.246667 2938 ts_tablet_manager.cc:1916] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.246985 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1206) took 4541 ms. Trace:
I20251029 08:52:34.247134 1974 rpcz_store.cc:276] 1029 08:52:29.705939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.705989 (+ 50us) service_pool.cc:225] Handling call
1029 08:52:34.246862 (+4540873us) ts_tablet_manager.cc:1151] Deleting tablet 8017e8fe5f634ef0bf7f8be7c8aea033
1029 08:52:34.246871 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.246949 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540717}
W20251029 08:52:34.247558 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 8017e8fe5f634ef0bf7f8be7c8aea033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8017e8fe5f634ef0bf7f8be7c8aea033 already in progress: creating tablet
I20251029 08:52:34.247964 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1201) took 4546 ms. Trace:
I20251029 08:52:34.249548 1974 rpcz_store.cc:276] 1029 08:52:29.701190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.712348 (+ 11158us) service_pool.cc:225] Handling call
1029 08:52:34.247832 (+4535484us) ts_tablet_manager.cc:1151] Deleting tablet b79ec90d93ee48abaaed3bc3cd41bf62
1029 08:52:34.247842 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.247934 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4534105}
W20251029 08:52:34.250015 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet b79ec90d93ee48abaaed3bc3cd41bf62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b79ec90d93ee48abaaed3bc3cd41bf62 already in progress: creating tablet
I20251029 08:52:34.250402 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1207) took 4535 ms. Trace:
I20251029 08:52:34.250538 1974 rpcz_store.cc:276] 1029 08:52:29.715278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.725069 (+ 9791us) service_pool.cc:225] Handling call
1029 08:52:34.250286 (+4525217us) ts_tablet_manager.cc:1151] Deleting tablet 3f25fe685db94f3ba2003bc7e417277c
1029 08:52:34.250296 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.250371 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4524903}
W20251029 08:52:34.250962 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3f25fe685db94f3ba2003bc7e417277c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f25fe685db94f3ba2003bc7e417277c already in progress: creating tablet
I20251029 08:52:34.251358 1974 tablet_replica.cc:333] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.251839 1176 ts_tablet_manager.cc:1929] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.252243 1974 raft_consensus.cc:2243] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.252979 1974 raft_consensus.cc:2272] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.252928 2920 raft_consensus.cc:493] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.253180 1176 log.cc:1199] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/a712185fe4aa4f8faf1ce2df229c11ee
I20251029 08:52:34.253191 2920 raft_consensus.cc:515] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.253530 1176 ts_tablet_manager.cc:1950] T a712185fe4aa4f8faf1ce2df229c11ee P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:34.253983 2920 leader_election.cc:290] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.254789 1974 ts_tablet_manager.cc:1916] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.255002 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a8c25ba4e9c410fa6d12949d6e595d7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:34.255544 720 leader_election.cc:343] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.257402 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 66329e1e548949cc9d59097953288375 (table default.Table_5c0b5dd5fcc24859812282551226c27e [id=7478238b5a424beaa652e44adb59822f]). Expected version 1 got 0
I20251029 08:52:34.261230 1176 tablet_replica.cc:333] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: stopping tablet replica
I20251029 08:52:34.261451 1176 raft_consensus.cc:2243] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.261612 1176 raft_consensus.cc:2272] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.262002 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f67e7b28b4240618e1817bb47d5633d with delete_type TABLET_DATA_DELETED (replaced by tablet 8bc9ac40dd5b49578945e3e68e235c04) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.262192 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 65eaf12e45c54bd1b0aedc647dc90436 with delete_type TABLET_DATA_DELETED (replaced by tablet 295ab2a10a5648abb59bc045c7dbe165) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.262609 1177 tablet_bootstrap.cc:492] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.262804 1177 ts_tablet_manager.cc:1403] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.003s
I20251029 08:52:34.263640 1177 raft_consensus.cc:359] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.263931 1177 raft_consensus.cc:385] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.264077 1177 raft_consensus.cc:740] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.264348 1177 consensus_queue.cc:260] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.264519 1183 tablet_bootstrap.cc:654] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.267248 1177 ts_tablet_manager.cc:1434] T fcaa3fa779824b38887ae001623ba67b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 08:52:34.267596 1177 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.269304 1177 tablet_bootstrap.cc:654] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.271831 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet a712185fe4aa4f8faf1ce2df229c11ee (table default.Table_e857915b93d94731a6672caf050b7807 [id=c0f3daf63db74c0e8d27140fd8d40c23]) successfully deleted
W20251029 08:52:34.269892 861 leader_election.cc:343] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 0bd490da0b0e47fc965343c123a27b22
I20251029 08:52:34.273664 861 leader_election.cc:304] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.273911 923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7a6008b4e6243ccaf166a31f1a9f62c" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.274153 893 tablet_service.cc:1552] Processing DeleteTablet for tablet fcaa3fa779824b38887ae001623ba67b with delete_type TABLET_DATA_DELETED (replaced by tablet 919e5494b84343639a2b9b81b05f1c2c) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.274338 2967 raft_consensus.cc:2749] T 0bd490da0b0e47fc965343c123a27b22 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.271816 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a8c25ba4e9c410fa6d12949d6e595d7" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.262494 1176 ts_tablet_manager.cc:1916] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.281265 2920 raft_consensus.cc:493] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.281395 2920 raft_consensus.cc:515] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.282024 2920 leader_election.cc:290] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.282696 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57c0ad83e3f54150850b6617a1781382" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:34.283228 720 leader_election.cc:343] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
W20251029 08:52:34.284178 998 leader_election.cc:343] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 65eaf12e45c54bd1b0aedc647dc90436
I20251029 08:52:34.284335 998 leader_election.cc:304] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730d0f5b7abf43a293b7df13d01ce4ee; no voters: 7a8d80098fcb40ada903c3b2bacc2ea0, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.284614 3003 raft_consensus.cc:2749] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.285316 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57c0ad83e3f54150850b6617a1781382" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:34.285988 718 leader_election.cc:343] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.286105 718 leader_election.cc:304] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
W20251029 08:52:34.286288 718 raft_consensus.cc:2686] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 08:52:34.286576 718 leader_election.cc:343] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.286609 2938 ts_tablet_manager.cc:1929] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.286706 2938 log.cc:1199] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/58932f38c0d64aec9a02238eaea56a25
I20251029 08:52:34.286679 718 leader_election.cc:304] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.286895 2920 raft_consensus.cc:2749] T 6a8c25ba4e9c410fa6d12949d6e595d7 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.287017 2938 ts_tablet_manager.cc:1950] T 58932f38c0d64aec9a02238eaea56a25 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
W20251029 08:52:34.287055 718 leader_election.cc:343] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.287145 718 leader_election.cc:304] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.287343 2920 raft_consensus.cc:2749] T 57c0ad83e3f54150850b6617a1781382 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.287463 2920 raft_consensus.cc:493] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.287546 2920 raft_consensus.cc:515] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.288069 2920 leader_election.cc:290] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.288676 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47470a907c1949989f79e8a2b803dc25" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:34.288695 2967 raft_consensus.cc:493] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.288856 2967 raft_consensus.cc:515] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
W20251029 08:52:34.289251 720 leader_election.cc:343] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.289407 2967 leader_election.cc:290] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969), 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031)
W20251029 08:52:34.290495 861 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 9429f6abb50c4f66b2e1b565f303962d
W20251029 08:52:34.290707 861 leader_election.cc:343] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 9429f6abb50c4f66b2e1b565f303962d
I20251029 08:52:34.290797 861 leader_election.cc:304] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.291250 2967 raft_consensus.cc:2749] T 9429f6abb50c4f66b2e1b565f303962d P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.291376 1974 ts_tablet_manager.cc:1929] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.295838 2938 tablet_replica.cc:333] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.295867 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 58932f38c0d64aec9a02238eaea56a25 (table default.Table_57929518185c4558a0f9f12c40945be2 [id=60ed38dd0d6e4bc5851de7c04b92abfc]) successfully deleted
I20251029 08:52:34.296461 2938 raft_consensus.cc:2243] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.296751 1183 tablet_bootstrap.cc:492] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.297014 1183 ts_tablet_manager.cc:1403] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.000s
I20251029 08:52:34.297997 1183 raft_consensus.cc:359] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.298318 1183 raft_consensus.cc:385] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.298441 1183 raft_consensus.cc:740] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.298784 1183 consensus_queue.cc:260] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.300148 595 catalog_manager.cc:5741] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969) does not have the latest schema for tablet 57c0ad83e3f54150850b6617a1781382 (table default.Table_f861cefe281a475fbd4ef5e56a0f7b8f [id=133929f2584345a98d53db9c2e42103c]). Expected version 2 got 0
I20251029 08:52:34.301457 1183 ts_tablet_manager.cc:1434] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:34.301746 1183 tablet_bootstrap.cc:492] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.303490 1183 tablet_bootstrap.cc:654] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.296741 2938 raft_consensus.cc:2272] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.304615 1974 log.cc:1199] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/6689cf0804f946c18ebc555a191549d4
I20251029 08:52:34.305588 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 308da528313044929b66040d6659f774 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:16 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.306596 1176 ts_tablet_manager.cc:1929] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.306707 1176 log.cc:1199] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-0/wal/wals/b7a6008b4e6243ccaf166a31f1a9f62c
I20251029 08:52:34.306991 1176 ts_tablet_manager.cc:1950] T b7a6008b4e6243ccaf166a31f1a9f62c P 7a8d80098fcb40ada903c3b2bacc2ea0: Deleting consensus metadata
I20251029 08:52:34.308413 2938 ts_tablet_manager.cc:1916] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.308856 1974 ts_tablet_manager.cc:1950] T 6689cf0804f946c18ebc555a191549d4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
W20251029 08:52:34.310454 583 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ad8a88123f8542eea86d8656bfef97df (table default.Table_07159bd6d35d4ba3b84c2c8c27ac779b [id=7a37f6e30a304e17a7a0ab5bd6b9564d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ad8a88123f8542eea86d8656bfef97df consensus state has no leader
I20251029 08:52:34.311486 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1209) took 4546 ms. Trace:
I20251029 08:52:34.311673 1974 rpcz_store.cc:276] 1029 08:52:29.765248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.765507 (+ 259us) service_pool.cc:225] Handling call
1029 08:52:34.251241 (+4485734us) ts_tablet_manager.cc:1151] Deleting tablet 6689cf0804f946c18ebc555a191549d4
1029 08:52:34.251250 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.261075 (+ 9825us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.291256 (+ 30181us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.311441 (+ 20185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4485510}
I20251029 08:52:34.312608 1974 tablet_replica.cc:333] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.313297 1974 raft_consensus.cc:2243] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:34.314800 1974 pending_rounds.cc:70] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:34.315048 1974 pending_rounds.cc:77] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037685279666176 op_type: NO_OP noop_request { }
I20251029 08:52:34.315591 3032 raft_consensus.cc:493] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.315625 1974 raft_consensus.cc:2889] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:34.315821 3032 raft_consensus.cc:515] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.316224 1974 raft_consensus.cc:2272] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.316692 3032 leader_election.cc:290] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433)
I20251029 08:52:34.317106 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87c98206d7fa409b8017abc72688db93" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
W20251029 08:52:34.317651 720 leader_election.cc:343] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.319034 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47470a907c1949989f79e8a2b803dc25" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.319639 584 catalog_manager.cc:4985] TS 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): tablet b7a6008b4e6243ccaf166a31f1a9f62c (table default.Table_d6e32e655391437ea03d14eade0cc35f [id=ab4b38f7d715445b80c726527cc3419a]) successfully deleted
I20251029 08:52:34.320369 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87c98206d7fa409b8017abc72688db93" candidate_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
I20251029 08:52:34.320647 1183 tablet_bootstrap.cc:492] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.320914 1183 ts_tablet_manager.cc:1403] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251029 08:52:34.321731 1183 raft_consensus.cc:359] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.321967 1183 raft_consensus.cc:385] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.322059 1183 raft_consensus.cc:740] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.322333 1183 consensus_queue.cc:260] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.327358 759 tablet_service.cc:1552] Processing DeleteTablet for tablet 633ace1413464e939e4e47782370e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:39982
W20251029 08:52:34.327611 718 leader_election.cc:343] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.327621 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ee033a724942b0a90f12f15cba2701 with delete_type TABLET_DATA_DELETED (replaced by tablet a013257089774ba8b1f535e5f94ac1bb) from {username='slave'} at 127.0.0.1:39982
I20251029 08:52:34.327920 718 leader_election.cc:304] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8d80098fcb40ada903c3b2bacc2ea0; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 927689e0dc5e4bc79d4a9d79a873f088
I20251029 08:52:34.327790 824 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 08:52:34.327823 3032 raft_consensus.cc:493] T 85ee033a724942b0a90f12f15cba2701 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.328286 2920 raft_consensus.cc:2749] T 47470a907c1949989f79e8a2b803dc25 P 7a8d80098fcb40ada903c3b2bacc2ea0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 08:52:34.328711 718 leader_election.cc:343] T 87c98206d7fa409b8017abc72688db93 P 7a8d80098fcb40ada903c3b2bacc2ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.328891 2967 raft_consensus.cc:493] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.329035 2967 raft_consensus.cc:515] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.329685 2967 leader_election.cc:290] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.330144 1183 ts_tablet_manager.cc:1434] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
W20251029 08:52:34.330922 861 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 7046bb1fcdd34d308aecddfdf0752911
W20251029 08:52:34.331578 861 leader_election.cc:343] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: 7046bb1fcdd34d308aecddfdf0752911
I20251029 08:52:34.331688 861 leader_election.cc:304] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.332144 2967 raft_consensus.cc:2749] T 7046bb1fcdd34d308aecddfdf0752911 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.312278 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 6689cf0804f946c18ebc555a191549d4 (table default.Table_a565b344ef8e4b60b07cb259f877df2a [id=bef1fc3338f941f18e6ce2019247b13e]) successfully deleted
I20251029 08:52:34.336629 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cac145320cf48b4aaf4adb0b686ba8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:15 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.337044 1183 tablet_bootstrap.cc:492] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.338825 1183 tablet_bootstrap.cc:654] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.341418 1974 ts_tablet_manager.cc:1916] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.347105 2938 ts_tablet_manager.cc:1929] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.348126 2938 log.cc:1199] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/9499d0099d1c4edc843fb8d29ca3b1b6
I20251029 08:52:34.349151 2938 ts_tablet_manager.cc:1950] T 9499d0099d1c4edc843fb8d29ca3b1b6 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.350528 2938 tablet_replica.cc:333] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.350680 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 9499d0099d1c4edc843fb8d29ca3b1b6 (table default.Table_c407207334e644189fdbfcae1dc12ba9 [id=e1fb342ea0a541ce9f668a9410db6487]) successfully deleted
I20251029 08:52:34.351652 2938 raft_consensus.cc:2243] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.352169 2938 raft_consensus.cc:2272] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.352927 1177 tablet_bootstrap.cc:492] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.353168 1177 ts_tablet_manager.cc:1403] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20251029 08:52:34.354107 1177 raft_consensus.cc:359] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.354580 2967 raft_consensus.cc:493] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.354739 2967 raft_consensus.cc:515] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.355397 2967 leader_election.cc:290] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:34.356779 861 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: b8f2b34642704cd6bb966b6dcd494d51
W20251029 08:52:34.357458 861 leader_election.cc:343] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969): Not found: Tablet not found: b8f2b34642704cd6bb966b6dcd494d51
I20251029 08:52:34.357584 861 leader_election.cc:304] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927689e0dc5e4bc79d4a9d79a873f088; no voters: 730d0f5b7abf43a293b7df13d01ce4ee, 7a8d80098fcb40ada903c3b2bacc2ea0
I20251029 08:52:34.358038 2967 raft_consensus.cc:2749] T b8f2b34642704cd6bb966b6dcd494d51 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 08:52:34.358080 1974 ts_tablet_manager.cc:1929] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.358203 1974 log.cc:1199] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/0ab56045bd814d59831a98618c759af4
I20251029 08:52:34.354389 1177 raft_consensus.cc:385] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.358596 1177 raft_consensus.cc:740] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.358999 1177 consensus_queue.cc:260] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.368744 2938 ts_tablet_manager.cc:1916] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.373626 893 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b52314139945dd9c2e6ec899bf06dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.375300 1177 ts_tablet_manager.cc:1434] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251029 08:52:34.382494 1177 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.384470 1177 tablet_bootstrap.cc:654] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.386353 1183 tablet_bootstrap.cc:492] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.386508 1183 ts_tablet_manager.cc:1403] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.003s
I20251029 08:52:34.387135 1183 raft_consensus.cc:359] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.387279 1183 raft_consensus.cc:385] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.387337 1183 raft_consensus.cc:740] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.387553 1183 consensus_queue.cc:260] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.390092 3003 raft_consensus.cc:493] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.390221 3003 raft_consensus.cc:515] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.390794 3003 leader_election.cc:290] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.391456 921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c90087c9e94665a9b3fac798304af7" candidate_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927689e0dc5e4bc79d4a9d79a873f088" is_pre_election: true
W20251029 08:52:34.393004 996 leader_election.cc:343] T 59c90087c9e94665a9b3fac798304af7 P 730d0f5b7abf43a293b7df13d01ce4ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Illegal state: must be running to vote when last-logged opid is not known
I20251029 08:52:34.395586 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 00829c7fa0fe4e379a1a3044550e5941 with delete_type TABLET_DATA_DELETED (replaced by tablet 23aa4288b74441168088245c52915764) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.396247 1183 ts_tablet_manager.cc:1434] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 08:52:34.398840 1183 tablet_bootstrap.cc:492] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.402547 1974 ts_tablet_manager.cc:1950] T 0ab56045bd814d59831a98618c759af4 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.402628 2938 ts_tablet_manager.cc:1929] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.404844 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1210) took 4639 ms. Trace:
I20251029 08:52:34.404968 1974 rpcz_store.cc:276] 1029 08:52:29.765344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.765773 (+ 429us) service_pool.cc:225] Handling call
1029 08:52:34.312481 (+4546708us) ts_tablet_manager.cc:1151] Deleting tablet 0ab56045bd814d59831a98618c759af4
1029 08:52:34.312493 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.354703 (+ 42210us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.357944 (+ 3241us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.404799 (+ 46855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4546521}
I20251029 08:52:34.405424 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 0ab56045bd814d59831a98618c759af4 (table default.Table_29abde71edfc496a8c61702f28f25c63 [id=771dd3dd3a224a8599ab86bc7801dfa3]) successfully deleted
I20251029 08:52:34.406248 1974 tablet_replica.cc:333] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.406708 1974 raft_consensus.cc:2243] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.406939 1974 raft_consensus.cc:2272] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.407467 1974 ts_tablet_manager.cc:1916] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.406034 1183 tablet_bootstrap.cc:654] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.412161 2938 log.cc:1199] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/42de32d87ebb451090e0310e36e64277
I20251029 08:52:34.412642 2938 ts_tablet_manager.cc:1950] T 42de32d87ebb451090e0310e36e64277 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.412659 1177 tablet_bootstrap.cc:492] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.412863 1177 ts_tablet_manager.cc:1403] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251029 08:52:34.413713 2938 tablet_replica.cc:333] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.413900 2938 raft_consensus.cc:2243] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.414043 2938 raft_consensus.cc:2272] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.414024 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 42de32d87ebb451090e0310e36e64277 (table default.Table_007a17f442c74a71a6bb3775642f35fd [id=3ca17547257d4c358a173ebef3ef4a43]) successfully deleted
I20251029 08:52:34.415165 1177 raft_consensus.cc:359] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.415455 1177 raft_consensus.cc:385] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.415608 1177 raft_consensus.cc:740] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.415961 1177 consensus_queue.cc:260] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.416671 1177 ts_tablet_manager.cc:1434] T 2dec33b502aa4916b89b878462de1ab1 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 08:52:34.415302 1974 ts_tablet_manager.cc:1929] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.416896 1974 log.cc:1199] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/254e62d88c3242a0a3ac7470a8005d74
I20251029 08:52:34.417280 1974 ts_tablet_manager.cc:1950] T 254e62d88c3242a0a3ac7470a8005d74 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.418305 2938 ts_tablet_manager.cc:1916] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.424083 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:34.426362 2938 ts_tablet_manager.cc:1929] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.426486 2938 log.cc:1199] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/5f67e7b28b4240618e1817bb47d5633d
I20251029 08:52:34.426756 2938 ts_tablet_manager.cc:1950] T 5f67e7b28b4240618e1817bb47d5633d P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.427845 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 5f67e7b28b4240618e1817bb47d5633d (table default.Table_f38c5c596ef148eb8ac7ea611a4e4b3d [id=66eee56ac86141938732092093b9e0b2]) successfully deleted
I20251029 08:52:34.431998 1183 tablet_bootstrap.cc:492] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.432195 1183 ts_tablet_manager.cc:1403] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.008s
I20251029 08:52:34.432968 1183 raft_consensus.cc:359] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.433143 1183 raft_consensus.cc:385] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.433193 1183 raft_consensus.cc:740] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.433408 1183 consensus_queue.cc:260] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.438097 1183 ts_tablet_manager.cc:1434] T 527a83465a844849811b0f9d26428af0 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 08:52:34.438439 1183 tablet_bootstrap.cc:492] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.440251 1183 tablet_bootstrap.cc:654] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.441319 2938 tablet_replica.cc:333] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.441510 2938 raft_consensus.cc:2243] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.441689 2938 raft_consensus.cc:2272] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.442575 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 527a83465a844849811b0f9d26428af0 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]). Expected version 1 got 0
I20251029 08:52:34.444084 2938 ts_tablet_manager.cc:1916] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.457418 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet 527a83465a844849811b0f9d26428af0 (table default.Table_4757cb1c3548493485ab674b57397de5 [id=80e7ea583ee14114878007a237b185c8]). Expected version 1 got 0
I20251029 08:52:34.465633 2938 ts_tablet_manager.cc:1929] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.473201 1177 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.474920 1177 tablet_bootstrap.cc:654] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.460923 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1211) took 4695 ms. Trace:
I20251029 08:52:34.477494 1974 rpcz_store.cc:276] 1029 08:52:29.765442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.765983 (+ 541us) service_pool.cc:225] Handling call
1029 08:52:34.406157 (+4640174us) ts_tablet_manager.cc:1151] Deleting tablet 254e62d88c3242a0a3ac7470a8005d74
1029 08:52:34.406172 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.409769 (+ 3597us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.415176 (+ 5407us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.460865 (+ 45689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4640072}
I20251029 08:52:34.478092 1974 tablet_replica.cc:333] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.478241 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 254e62d88c3242a0a3ac7470a8005d74 (table default.Table_ddd6b7bc3669413fb31ff9f3414d5d9f [id=e3eb4371e34b43289550f02ae21e54d1]) successfully deleted
I20251029 08:52:34.478341 1974 raft_consensus.cc:2243] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: Raft consensus shutting down.
I20251029 08:52:34.478709 1974 pending_rounds.cc:70] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Trying to abort 1 pending ops.
I20251029 08:52:34.478863 1974 pending_rounds.cc:77] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7216037682131070976 op_type: NO_OP noop_request { }
I20251029 08:52:34.479087 1974 raft_consensus.cc:2889] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 08:52:34.479262 1974 raft_consensus.cc:2272] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.480105 1974 ts_tablet_manager.cc:1916] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.480111 2938 log.cc:1199] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/65eaf12e45c54bd1b0aedc647dc90436
I20251029 08:52:34.480469 2938 ts_tablet_manager.cc:1950] T 65eaf12e45c54bd1b0aedc647dc90436 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.481626 2938 tablet_replica.cc:333] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.481593 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 65eaf12e45c54bd1b0aedc647dc90436 (table default.Table_564ecd352b5a4bca97c4f3dfecdaad0e [id=413ee8ef11c04c8980ad0b69d7ed44fa]) successfully deleted
I20251029 08:52:34.481868 2938 raft_consensus.cc:2243] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.482065 2938 raft_consensus.cc:2272] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.489289 2938 ts_tablet_manager.cc:1916] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.496996 2938 ts_tablet_manager.cc:1929] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:34.497017 996 consensus_peers.cc:597] T c32679cbb1884f09addb0f7c26ac941e P 730d0f5b7abf43a293b7df13d01ce4ee -> Peer 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): Couldn't send request to peer 927689e0dc5e4bc79d4a9d79a873f088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 08:52:34.497267 2938 log.cc:1199] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/308da528313044929b66040d6659f774
I20251029 08:52:34.497609 2938 ts_tablet_manager.cc:1950] T 308da528313044929b66040d6659f774 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.498858 2938 tablet_replica.cc:333] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.499006 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 308da528313044929b66040d6659f774 (table default.Table_9f4e595ea8564e9d902458aff13d5c78 [id=205b9176bb44494b9b2f384073458db8]) successfully deleted
I20251029 08:52:34.499138 2938 raft_consensus.cc:2243] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.499281 2938 raft_consensus.cc:2272] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.500512 2938 ts_tablet_manager.cc:1916] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.501652 1177 tablet_bootstrap.cc:492] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.501811 1177 ts_tablet_manager.cc:1403] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251029 08:52:34.502810 1177 raft_consensus.cc:359] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.502996 1177 raft_consensus.cc:385] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.503051 1177 raft_consensus.cc:740] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.501648 1183 tablet_bootstrap.cc:492] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.503315 1183 ts_tablet_manager.cc:1403] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.065s user 0.023s sys 0.016s
I20251029 08:52:34.504197 1183 raft_consensus.cc:359] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.507361 1974 ts_tablet_manager.cc:1929] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 08:52:34.507478 1974 log.cc:1199] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/d5ac1af8b78a4d94b0e3da14ebfffd92
I20251029 08:52:34.507582 1183 raft_consensus.cc:385] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.507700 1183 raft_consensus.cc:740] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.507864 1974 ts_tablet_manager.cc:1950] T d5ac1af8b78a4d94b0e3da14ebfffd92 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.508123 1177 consensus_queue.cc:260] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.508699 1177 ts_tablet_manager.cc:1434] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 08:52:34.513244 588 catalog_manager.cc:5741] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433) does not have the latest schema for tablet 2dec33b502aa4916b89b878462de1ab1 (table default.Table_4ca13bdf97f04056ba795f885976e736 [id=5f7ee394d987440c9ab0c7ad53bfebb4]). Expected version 2 got 0
I20251029 08:52:34.513590 1177 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.515143 1177 tablet_bootstrap.cc:654] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.508227 1183 consensus_queue.cc:260] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.515942 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cd2e9a95b6c40be9e17397a313d249b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:14 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.520551 1183 ts_tablet_manager.cc:1434] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251029 08:52:34.520915 1183 tablet_bootstrap.cc:492] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.523451 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fad7549758649229ad771de926aaf15 with delete_type TABLET_DATA_DELETED (replaced by tablet 173132d8c3af4b5a874f51bd8ba61b68) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.527591 1183 tablet_bootstrap.cc:654] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.535902 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1208) took 4770 ms. Trace:
I20251029 08:52:34.536044 1974 rpcz_store.cc:276] 1029 08:52:29.765150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.766224 (+ 1074us) service_pool.cc:225] Handling call
1029 08:52:34.478041 (+4711817us) ts_tablet_manager.cc:1151] Deleting tablet d5ac1af8b78a4d94b0e3da14ebfffd92
1029 08:52:34.478052 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.504099 (+ 26047us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.507276 (+ 3177us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.535873 (+ 28597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4710123}
I20251029 08:52:34.536466 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1218) took 4564 ms. Trace:
I20251029 08:52:34.536553 1974 rpcz_store.cc:276] 1029 08:52:29.972221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:29.973696 (+ 1475us) service_pool.cc:225] Handling call
1029 08:52:34.536334 (+4562638us) ts_tablet_manager.cc:1151] Deleting tablet ea40ed38c0ba4d2f9045188af226ced5
1029 08:52:34.536346 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.536449 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4562483}
I20251029 08:52:34.536854 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1219) took 4504 ms. Trace:
I20251029 08:52:34.536929 1974 rpcz_store.cc:276] 1029 08:52:30.032178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.032406 (+ 228us) service_pool.cc:225] Handling call
1029 08:52:34.536750 (+4504344us) ts_tablet_manager.cc:1151] Deleting tablet 307262da8888461ebddaaf33a31be8b8
1029 08:52:34.536758 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.536839 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4504181}
I20251029 08:52:34.537221 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1221) took 4504 ms. Trace:
I20251029 08:52:34.537290 1974 rpcz_store.cc:276] 1029 08:52:30.032350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.032491 (+ 141us) service_pool.cc:225] Handling call
1029 08:52:34.537119 (+4504628us) ts_tablet_manager.cc:1151] Deleting tablet 4524a248859448c5b789c7ae82435705
1029 08:52:34.537127 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.537209 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4504492}
I20251029 08:52:34.537564 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1220) took 4505 ms. Trace:
I20251029 08:52:34.537629 1974 rpcz_store.cc:276] 1029 08:52:30.032275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.032411 (+ 136us) service_pool.cc:225] Handling call
1029 08:52:34.537471 (+4505060us) ts_tablet_manager.cc:1151] Deleting tablet 4675f0aec6a14807afd3094780d60c72
1029 08:52:34.537479 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.537547 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4504631}
I20251029 08:52:34.537662 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet d5ac1af8b78a4d94b0e3da14ebfffd92 (table default.Table_5c54cf056b2f4d76865c7eba3b88f6f6 [id=940a120e677049d1851b2ba9671e3200]) successfully deleted
I20251029 08:52:34.537918 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1222) took 4502 ms. Trace:
W20251029 08:52:34.537951 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet ea40ed38c0ba4d2f9045188af226ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea40ed38c0ba4d2f9045188af226ced5 already in progress: creating tablet
I20251029 08:52:34.537985 1974 rpcz_store.cc:276] 1029 08:52:30.035151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.037354 (+ 2203us) service_pool.cc:225] Handling call
1029 08:52:34.537831 (+4500477us) ts_tablet_manager.cc:1151] Deleting tablet f2c18e4c41714ba38bdd2e208a5e761e
1029 08:52:34.537839 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.537904 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4500350}
W20251029 08:52:34.538136 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 307262da8888461ebddaaf33a31be8b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 307262da8888461ebddaaf33a31be8b8 already in progress: creating tablet
I20251029 08:52:34.538244 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1226) took 4338 ms. Trace:
W20251029 08:52:34.538324 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4524a248859448c5b789c7ae82435705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4524a248859448c5b789c7ae82435705 already in progress: creating tablet
I20251029 08:52:34.538306 1974 rpcz_store.cc:276] 1029 08:52:30.199926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.200233 (+ 307us) service_pool.cc:225] Handling call
1029 08:52:34.538160 (+4337927us) ts_tablet_manager.cc:1151] Deleting tablet e27701fbf5b44e90ba33bed99fdc4d9b
1029 08:52:34.538169 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.538232 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4337730}
I20251029 08:52:34.538601 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1227) took 4338 ms. Trace:
I20251029 08:52:34.538664 1974 rpcz_store.cc:276] 1029 08:52:30.200188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.200348 (+ 160us) service_pool.cc:225] Handling call
1029 08:52:34.538508 (+4338160us) ts_tablet_manager.cc:1151] Deleting tablet 9e44e56554574d5abb67d08d88218152
1029 08:52:34.538519 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.538588 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4338048}
I20251029 08:52:34.538954 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1225) took 4345 ms. Trace:
I20251029 08:52:34.539021 1974 rpcz_store.cc:276] 1029 08:52:30.193026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.202269 (+ 9243us) service_pool.cc:225] Handling call
1029 08:52:34.538862 (+4336593us) ts_tablet_manager.cc:1151] Deleting tablet 3ee83f5526f74ba2a344edd97db94050
1029 08:52:34.538870 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.538939 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336394}
I20251029 08:52:34.539309 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1228) took 4254 ms. Trace:
I20251029 08:52:34.539372 1974 rpcz_store.cc:276] 1029 08:52:30.284454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.289426 (+ 4972us) service_pool.cc:225] Handling call
1029 08:52:34.539220 (+4249794us) ts_tablet_manager.cc:1151] Deleting tablet 0adba75cc7d74b1497de9af7ad7ccdd0
1029 08:52:34.539228 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.539293 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4249522}
I20251029 08:52:34.539675 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1236) took 3872 ms. Trace:
W20251029 08:52:34.539713 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4675f0aec6a14807afd3094780d60c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4675f0aec6a14807afd3094780d60c72 already in progress: creating tablet
I20251029 08:52:34.539750 1974 rpcz_store.cc:276] 1029 08:52:30.667401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.699073 (+ 31672us) service_pool.cc:225] Handling call
1029 08:52:34.539584 (+3840511us) ts_tablet_manager.cc:1151] Deleting tablet d87191d05d4145a796a8823f723bc499
1029 08:52:34.539592 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.539662 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3840303}
W20251029 08:52:34.539894 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet f2c18e4c41714ba38bdd2e208a5e761e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c18e4c41714ba38bdd2e208a5e761e already in progress: creating tablet
W20251029 08:52:34.540102 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e27701fbf5b44e90ba33bed99fdc4d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e27701fbf5b44e90ba33bed99fdc4d9b already in progress: creating tablet
W20251029 08:52:34.540289 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 9e44e56554574d5abb67d08d88218152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e44e56554574d5abb67d08d88218152 already in progress: creating tablet
W20251029 08:52:34.540477 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 3ee83f5526f74ba2a344edd97db94050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee83f5526f74ba2a344edd97db94050 already in progress: creating tablet
I20251029 08:52:34.541122 2938 ts_tablet_manager.cc:1929] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 08:52:34.541198 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 0adba75cc7d74b1497de9af7ad7ccdd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0adba75cc7d74b1497de9af7ad7ccdd0 already in progress: creating tablet
I20251029 08:52:34.541258 2938 log.cc:1199] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/3cac145320cf48b4aaf4adb0b686ba8c
W20251029 08:52:34.541409 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet d87191d05d4145a796a8823f723bc499 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d87191d05d4145a796a8823f723bc499 already in progress: creating tablet
I20251029 08:52:34.541607 2938 ts_tablet_manager.cc:1950] T 3cac145320cf48b4aaf4adb0b686ba8c P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.542503 2938 tablet_replica.cc:333] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.542618 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 3cac145320cf48b4aaf4adb0b686ba8c (table default.Table_43e66e8dbdf240a1b5eb4233ece857c7 [id=26886277edf44c6cbffbd904706dc430]) successfully deleted
I20251029 08:52:34.542698 2938 raft_consensus.cc:2243] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.542858 2938 raft_consensus.cc:2272] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.544718 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1237) took 3877 ms. Trace:
I20251029 08:52:34.544966 1974 rpcz_store.cc:276] 1029 08:52:30.667516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.667727 (+ 211us) service_pool.cc:225] Handling call
1029 08:52:34.539945 (+3872218us) ts_tablet_manager.cc:1151] Deleting tablet 7492195a41664ef69554d8a6cb148e21
1029 08:52:34.539953 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.544698 (+ 4745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3837458}
W20251029 08:52:34.545560 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 7492195a41664ef69554d8a6cb148e21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7492195a41664ef69554d8a6cb148e21 already in progress: creating tablet
I20251029 08:52:34.545977 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1240) took 3873 ms. Trace:
I20251029 08:52:34.546388 2967 raft_consensus.cc:493] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.546522 2967 raft_consensus.cc:515] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.547152 2967 leader_election.cc:290] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.547906 2938 ts_tablet_manager.cc:1916] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:34.548348 861 leader_election.cc:343] T e2b87d133f8a4bd1877e07a1a8758147 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: e2b87d133f8a4bd1877e07a1a8758147
I20251029 08:52:34.549638 1177 tablet_bootstrap.cc:492] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.549831 1177 ts_tablet_manager.cc:1403] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20251029 08:52:34.550508 1177 raft_consensus.cc:359] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.550740 1177 raft_consensus.cc:385] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.551000 1177 raft_consensus.cc:740] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.551299 1177 consensus_queue.cc:260] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.552294 1183 tablet_bootstrap.cc:492] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.552417 1183 ts_tablet_manager.cc:1403] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20251029 08:52:34.553172 1183 raft_consensus.cc:359] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.553339 1183 raft_consensus.cc:385] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.553380 1183 raft_consensus.cc:740] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.553578 1183 consensus_queue.cc:260] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.554044 1183 ts_tablet_manager.cc:1434] T a69beee4032f447eaa694d237cb0e4e3 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 08:52:34.558495 1177 ts_tablet_manager.cc:1434] T 0c2072c16fce43278f1e32b9672187e7 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251029 08:52:34.558880 1177 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.562311 2938 ts_tablet_manager.cc:1929] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.562546 2938 log.cc:1199] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/00829c7fa0fe4e379a1a3044550e5941
I20251029 08:52:34.562947 2938 ts_tablet_manager.cc:1950] T 00829c7fa0fe4e379a1a3044550e5941 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.563840 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c2072c16fce43278f1e32b9672187e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:25 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.564330 1183 tablet_bootstrap.cc:492] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.565997 1183 tablet_bootstrap.cc:654] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.566538 2967 raft_consensus.cc:493] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.566672 2967 raft_consensus.cc:515] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.567318 2967 leader_election.cc:290] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.569782 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a69beee4032f447eaa694d237cb0e4e3 (table default.Table_bb583028431740f4afd61a171a14298e [id=d687628169ff44f5b2cb137a0bc8e325]). Expected version 2 got 0
W20251029 08:52:34.572053 861 leader_election.cc:343] T 663d6587a5e14519afbb0f94d691d1b4 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 663d6587a5e14519afbb0f94d691d1b4
I20251029 08:52:34.573236 2938 tablet_replica.cc:333] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.573429 2938 raft_consensus.cc:2243] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.573582 2938 raft_consensus.cc:2272] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.576184 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 00829c7fa0fe4e379a1a3044550e5941 (table default.Table_19963a3f1dbd45ceab6c86ad0efe7377 [id=d370c516b7874d73b5deef41cb2511ee]) successfully deleted
I20251029 08:52:34.576921 2938 ts_tablet_manager.cc:1916] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.577437 588 catalog_manager.cc:5741] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031) does not have the latest schema for tablet a69beee4032f447eaa694d237cb0e4e3 (table default.Table_bb583028431740f4afd61a171a14298e [id=d687628169ff44f5b2cb137a0bc8e325]). Expected version 2 got 0
I20251029 08:52:34.580049 2967 raft_consensus.cc:493] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.580199 2967 raft_consensus.cc:515] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.580762 2967 leader_election.cc:290] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:34.581873 861 leader_election.cc:343] T 3cd2e9a95b6c40be9e17397a313d249b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: 3cd2e9a95b6c40be9e17397a313d249b
I20251029 08:52:34.586037 2938 ts_tablet_manager.cc:1929] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.586881 1183 tablet_bootstrap.cc:492] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.587123 1183 ts_tablet_manager.cc:1403] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251029 08:52:34.587867 1183 raft_consensus.cc:359] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.588075 1183 raft_consensus.cc:385] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.588128 1183 raft_consensus.cc:740] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.588322 1183 consensus_queue.cc:260] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.588840 1183 ts_tablet_manager.cc:1434] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 08:52:34.589138 1183 tablet_bootstrap.cc:492] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.590910 1183 tablet_bootstrap.cc:654] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.592489 2938 log.cc:1199] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/2fad7549758649229ad771de926aaf15
I20251029 08:52:34.592991 2938 ts_tablet_manager.cc:1950] T 2fad7549758649229ad771de926aaf15 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.593685 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9c29b93a643ec837169a20c59de33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:11 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.594583 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 2fad7549758649229ad771de926aaf15 (table default.Table_4a8284ff6d314bbc976db77d1cb42d29 [id=5a81245c63184c9b9f5246f17d6e3a4e]) successfully deleted
I20251029 08:52:34.595748 2938 tablet_replica.cc:333] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.596215 2938 raft_consensus.cc:2243] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.596693 2938 raft_consensus.cc:2272] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.597872 2938 ts_tablet_manager.cc:1916] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.601325 1177 tablet_bootstrap.cc:654] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.601833 1974 rpcz_store.cc:276] 1029 08:52:30.672240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.672291 (+ 51us) service_pool.cc:225] Handling call
1029 08:52:34.545872 (+3873581us) ts_tablet_manager.cc:1151] Deleting tablet 4dec9fdf1ae94ae687f30f89096ce89b
1029 08:52:34.545881 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.545962 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3837761}
W20251029 08:52:34.602481 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 4dec9fdf1ae94ae687f30f89096ce89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dec9fdf1ae94ae687f30f89096ce89b already in progress: creating tablet
I20251029 08:52:34.602597 1183 tablet_bootstrap.cc:492] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.602715 1183 ts_tablet_manager.cc:1403] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20251029 08:52:34.603063 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1241) took 3928 ms. Trace:
I20251029 08:52:34.603197 1974 rpcz_store.cc:276] 1029 08:52:30.674899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.674955 (+ 56us) service_pool.cc:225] Handling call
1029 08:52:34.602917 (+3927962us) ts_tablet_manager.cc:1151] Deleting tablet 07bf5d693be240f18bacccc81a6ea5d4
1029 08:52:34.602931 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.603043 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3894725}
I20251029 08:52:34.603493 1183 raft_consensus.cc:359] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.603662 1183 raft_consensus.cc:385] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.603714 1183 raft_consensus.cc:740] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
W20251029 08:52:34.603714 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 07bf5d693be240f18bacccc81a6ea5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07bf5d693be240f18bacccc81a6ea5d4 already in progress: creating tablet
I20251029 08:52:34.603953 1183 consensus_queue.cc:260] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.604127 1974 tablet_replica.cc:333] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.604406 1974 raft_consensus.cc:2243] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.604626 1974 raft_consensus.cc:2272] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.605192 1974 ts_tablet_manager.cc:1916] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.608389 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f1283f5a9245a599e5576a1e84b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:12 UTC) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.612640 1974 ts_tablet_manager.cc:1929] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.615952 1183 ts_tablet_manager.cc:1434] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251029 08:52:34.616364 1183 tablet_bootstrap.cc:492] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.617115 1974 log.cc:1199] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/09968d93203140fa9201928c87702ee0
I20251029 08:52:34.617594 1974 ts_tablet_manager.cc:1950] T 09968d93203140fa9201928c87702ee0 P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.616961 1177 tablet_bootstrap.cc:492] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088: No bootstrap required, opened a new log
I20251029 08:52:34.617841 1177 ts_tablet_manager.cc:1403] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.000s
I20251029 08:52:34.618237 1183 tablet_bootstrap.cc:654] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.618680 1177 raft_consensus.cc:359] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.618918 1177 raft_consensus.cc:385] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.619024 1177 raft_consensus.cc:740] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927689e0dc5e4bc79d4a9d79a873f088, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.619318 1177 consensus_queue.cc:260] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.630767 2938 ts_tablet_manager.cc:1929] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.630908 2938 log.cc:1199] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/49d9c29b93a643ec837169a20c59de33
I20251029 08:52:34.631584 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 59ad04f57c29407b9d2420520c095a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 08:52:26 UTC) from {username='slave'} at 127.0.0.1:50036
I20251029 08:52:34.633091 2938 ts_tablet_manager.cc:1950] T 49d9c29b93a643ec837169a20c59de33 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.633416 1177 ts_tablet_manager.cc:1434] T 59ad04f57c29407b9d2420520c095a28 P 927689e0dc5e4bc79d4a9d79a873f088: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251029 08:52:34.634220 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet 49d9c29b93a643ec837169a20c59de33 (table default.Table_d1c5d76d3bd24eca923ed08a72bbc56b [id=af35c0b34122466c9e057ffde58e8836]) successfully deleted
I20251029 08:52:34.637362 2967 raft_consensus.cc:493] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.637517 2967 raft_consensus.cc:515] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.638144 2967 leader_election.cc:290] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
W20251029 08:52:34.639282 861 leader_election.cc:343] T f5b52314139945dd9c2e6ec899bf06dc P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: f5b52314139945dd9c2e6ec899bf06dc
I20251029 08:52:34.639832 2938 tablet_replica.cc:333] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.640704 2938 raft_consensus.cc:2243] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.641266 2938 raft_consensus.cc:2272] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.642405 2938 ts_tablet_manager.cc:1916] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.648255 1177 tablet_bootstrap.cc:492] T b1aaf233eba2461f9182e094a0115a4e P 927689e0dc5e4bc79d4a9d79a873f088: Bootstrap starting.
I20251029 08:52:34.649242 1183 tablet_bootstrap.cc:492] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.649405 1183 ts_tablet_manager.cc:1403] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20251029 08:52:34.650202 1183 raft_consensus.cc:359] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.650393 1183 raft_consensus.cc:385] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.650449 1183 raft_consensus.cc:740] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.650665 1183 consensus_queue.cc:260] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.651258 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1242) took 3968 ms. Trace:
I20251029 08:52:34.651352 1974 rpcz_store.cc:276] 1029 08:52:30.683029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.683226 (+ 197us) service_pool.cc:225] Handling call
1029 08:52:34.604077 (+3920851us) ts_tablet_manager.cc:1151] Deleting tablet 09968d93203140fa9201928c87702ee0
1029 08:52:34.604088 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.609418 (+ 5330us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.612536 (+ 3118us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.651230 (+ 38694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895848}
I20251029 08:52:34.651846 2967 raft_consensus.cc:493] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.651958 2967 raft_consensus.cc:515] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.652594 2967 leader_election.cc:290] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.652973 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 09968d93203140fa9201928c87702ee0 (table default.Table_06f02760c1014a1dbe9f6eea969646a9 [id=030256152a5942428f71542493eb8c0d]) successfully deleted
I20251029 08:52:34.653913 1177 tablet_bootstrap.cc:654] T b1aaf233eba2461f9182e094a0115a4e P 927689e0dc5e4bc79d4a9d79a873f088: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.655138 1974 tablet_replica.cc:333] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.655330 1974 raft_consensus.cc:2243] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.655488 1974 raft_consensus.cc:2272] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.656090 1974 ts_tablet_manager.cc:1916] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 08:52:34.656313 861 leader_election.cc:343] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): Not found: Tablet not found: bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:34.658674 1183 ts_tablet_manager.cc:1434] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 08:52:34.659065 1183 tablet_bootstrap.cc:492] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee: Bootstrap starting.
I20251029 08:52:34.660833 1183 tablet_bootstrap.cc:654] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee: Neither blocks nor log segments found. Creating new log.
I20251029 08:52:34.662256 2967 raft_consensus.cc:493] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 08:52:34.662382 2967 raft_consensus.cc:515] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } }
I20251029 08:52:34.662986 2967 leader_election.cc:290] T 66329e1e548949cc9d59097953288375 P 927689e0dc5e4bc79d4a9d79a873f088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031), 7a8d80098fcb40ada903c3b2bacc2ea0 (127.0.104.1:40969)
I20251029 08:52:34.663727 1065 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66329e1e548949cc9d59097953288375" candidate_uuid: "927689e0dc5e4bc79d4a9d79a873f088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" is_pre_election: true
I20251029 08:52:34.665558 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet d0d00a6b772d4034ac7fa764884c7081 with delete_type TABLET_DATA_DELETED (replaced by tablet e57e4313adce4e35843ccd9ee137e8f7) from {username='slave'} at 127.0.0.1:37708
I20251029 08:52:34.669289 1974 ts_tablet_manager.cc:1929] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.671603 1974 log.cc:1199] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/bf29e030892d44e1ad4d1ca4f5aeab3b
I20251029 08:52:34.672221 1974 ts_tablet_manager.cc:1950] T bf29e030892d44e1ad4d1ca4f5aeab3b P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.673136 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1243) took 3990 ms. Trace:
I20251029 08:52:34.673830 1974 rpcz_store.cc:276] 1029 08:52:30.683155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.683399 (+ 244us) service_pool.cc:225] Handling call
1029 08:52:34.655080 (+3971681us) ts_tablet_manager.cc:1151] Deleting tablet bf29e030892d44e1ad4d1ca4f5aeab3b
1029 08:52:34.655093 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.659822 (+ 4729us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.669142 (+ 9320us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.673116 (+ 3974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3946813}
I20251029 08:52:34.674623 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet bf29e030892d44e1ad4d1ca4f5aeab3b (table default.Table_0622352c737244ae90d1a8ac1a960864 [id=47176871ac0e427cab06027f096f666e]) successfully deleted
I20251029 08:52:34.675040 1974 tablet_replica.cc:333] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: stopping tablet replica
I20251029 08:52:34.675438 1974 raft_consensus.cc:2243] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.675907 1974 raft_consensus.cc:2272] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.675920 2938 ts_tablet_manager.cc:1929] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.676065 2938 log.cc:1199] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/b8f1283f5a9245a599e5576a1e84b4a5
I20251029 08:52:34.676407 2938 ts_tablet_manager.cc:1950] T b8f1283f5a9245a599e5576a1e84b4a5 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.677019 1974 ts_tablet_manager.cc:1916] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.677564 584 catalog_manager.cc:4985] TS 730d0f5b7abf43a293b7df13d01ce4ee (127.0.104.3:38031): tablet b8f1283f5a9245a599e5576a1e84b4a5 (table default.Table_183dac6abe3b4b6d8ef9614f941af3a0 [id=a99e94132c4e414ab2722fcb747ad901]) successfully deleted
I20251029 08:52:34.680934 2938 tablet_replica.cc:333] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: stopping tablet replica
I20251029 08:52:34.681195 2938 raft_consensus.cc:2243] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 08:52:34.681375 2938 raft_consensus.cc:2272] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 08:52:34.682732 1183 tablet_bootstrap.cc:492] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee: No bootstrap required, opened a new log
I20251029 08:52:34.682891 1183 ts_tablet_manager.cc:1403] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20251029 08:52:34.683908 1183 raft_consensus.cc:359] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.684151 1183 raft_consensus.cc:385] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 08:52:34.684212 1183 raft_consensus.cc:740] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d0f5b7abf43a293b7df13d01ce4ee, State: Initialized, Role: FOLLOWER
I20251029 08:52:34.684438 1183 consensus_queue.cc:260] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8d80098fcb40ada903c3b2bacc2ea0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40969 } } peers { permanent_uuid: "927689e0dc5e4bc79d4a9d79a873f088" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42433 } } peers { permanent_uuid: "730d0f5b7abf43a293b7df13d01ce4ee" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38031 } }
I20251029 08:52:34.688505 2938 ts_tablet_manager.cc:1916] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 08:52:34.690893 1974 ts_tablet_manager.cc:1929] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.692679 1974 log.cc:1199] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-1/wal/wals/657cdaad97e24611a997e26a6293ea5e
I20251029 08:52:34.693306 1974 ts_tablet_manager.cc:1950] T 657cdaad97e24611a997e26a6293ea5e P 927689e0dc5e4bc79d4a9d79a873f088: Deleting consensus metadata
I20251029 08:52:34.694928 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1244) took 3950 ms. Trace:
I20251029 08:52:34.695230 2938 ts_tablet_manager.cc:1929] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 08:52:34.695349 2938 log.cc:1199] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting WAL directory at /tmp/dist-test-taskLKoUQR/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761727917652956-416-0/minicluster-data/ts-2/wal/wals/d0d00a6b772d4034ac7fa764884c7081
I20251029 08:52:34.695677 2938 ts_tablet_manager.cc:1950] T d0d00a6b772d4034ac7fa764884c7081 P 730d0f5b7abf43a293b7df13d01ce4ee: Deleting consensus metadata
I20251029 08:52:34.696097 1974 rpcz_store.cc:276] 1029 08:52:30.744388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.744440 (+ 52us) service_pool.cc:225] Handling call
1029 08:52:34.674986 (+3930546us) ts_tablet_manager.cc:1151] Deleting tablet 657cdaad97e24611a997e26a6293ea5e
1029 08:52:34.674998 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.685268 (+ 10270us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.690778 (+ 5510us) tablet_metadata.cc:641] Metadata flushed
1029 08:52:34.694905 (+ 4127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3930265}
I20251029 08:52:34.696935 582 catalog_manager.cc:4985] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): tablet 657cdaad97e24611a997e26a6293ea5e (table default.Table_2576901b40a44a1a892f8bf476ee9519 [id=40d62f7b6ed74484a47a175de461b281]) successfully deleted
I20251029 08:52:34.697485 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1255) took 3858 ms. Trace:
I20251029 08:52:34.697933 1183 ts_tablet_manager.cc:1434] T 85a806cc04e641d9a1c4af97a24b13de P 730d0f5b7abf43a293b7df13d01ce4ee: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251029 08:52:34.697964 1974 rpcz_store.cc:276] 1029 08:52:30.838608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.838669 (+ 61us) service_pool.cc:225] Handling call
1029 08:52:34.697340 (+3858671us) ts_tablet_manager.cc:1151] Deleting tablet 6dc6c8f9ee9840c08c9436ab516a62d7
1029 08:52:34.697352 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.697466 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3843748}
W20251029 08:52:34.698765 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet 6dc6c8f9ee9840c08c9436ab516a62d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc6c8f9ee9840c08c9436ab516a62d7 already in progress: creating tablet
I20251029 08:52:34.700420 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1254) took 3861 ms. Trace:
I20251029 08:52:34.700827 1974 rpcz_store.cc:276] 1029 08:52:30.838510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:30.839923 (+ 1413us) service_pool.cc:225] Handling call
1029 08:52:34.700293 (+3860370us) ts_tablet_manager.cc:1151] Deleting tablet e322478c9ba64c2a95a2c299123926dd
1029 08:52:34.700305 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.700403 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3845188}
W20251029 08:52:34.701525 582 catalog_manager.cc:4965] TS 927689e0dc5e4bc79d4a9d79a873f088 (127.0.104.2:42433): delete failed for tablet e322478c9ba64c2a95a2c299123926dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e322478c9ba64c2a95a2c299123926dd already in progress: creating tablet
I20251029 08:52:34.702023 1974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50036 (request call id 1260) took 3714 ms. Trace:
I20251029 08:52:34.702433 1974 rpcz_store.cc:276] 1029 08:52:30.987732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 08:52:31.012095 (+ 24363us) service_pool.cc:225] Handling call
1029 08:52:34.701897 (+3689802us) ts_tablet_manager.cc:1151] Deleting tablet 0b0f1cdfb1804d1c8d6220907795fc80
1029 08:52:34.701908 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 08:52:34.702006 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us"